Do you work with lots of software and need to know what it's all getting up to, when you need to know it? I work for a .Net house with over 10 years' worth of software under our belts, we totally have this problem.
Our full estate includes many small and large systems - Web services, Windows services, .Net web apps, MVC web apps... if it's Microsoft badged, we'll have one somewhere. Each of these systems currently logs to a mixture of database tables, flat text files, e-mail alerts - the whole thing is becoming increasingly difficult to manage.
This blog post will be the first in a series looking at an open source solution, involving Log4Net, RabbitMq, Graylog2, ElasticSearch and NEsper, with the following capabilities:
- Aggregating logs from various distributed systems into a flexible and searchable persistent storage mechanism
- A web front-end for querying and graphing the log data
- A system capable of detecting abnormal or critical log patterns and alerting support staff