Logging Application

Logs are a critical part of any system, they give you deep insights about your application, what your system is doing and what caused the error, when something wrong happens. Virtually every system generates logs in some form or another, these logs are written to files on local disks. When you’re building enterprise level application, your system goes to multiple hosts, managing the logs across multiple hosts can be complicated. Debugging the error in the application across hundreds of log files on hundreds of servers can be very time consuming and complicated.

Challenges of Managing Logs Across Multiple Hosts

A common approach to this problem is building a centralized logging application which can collect and aggregate different types of logs in one central location. There are many tools available to which can solve some part of the problem but we need to build a robust application using all these tools. There are total four parts in centralized logging application — Collect logs, transport, store and analyse. We are going to look at each of this parts in depth and see how we can build an application.

Log Collection Strategies

All the applications create logs in different ways, some applications log through syslogs and other logs directly in files. When you see a typical web application running on a Linux server, there will be a dozen of more log files in /var/log and also a few application-specific logs in the home directories and other locations. Basically, there will be logs generated by different applications at a different place. Now, consider you have a web application running on the server and if something goes down, your developers or operations team need to access log data quickly in order to troubleshoot live issues, you would need a solution which can monitor the changes in the log files in almost real-time. To solve this issue, you can follow replication approach,

Replication Approach

In the replication approach, files are replicated to a central server on a fixed schedule. You will setup a cron job that will replicate your files on Linux server to your central server. A one-minute cron job might not be fast enough to troubleshoot when your site is down and you will be waiting for the relevant log data to be replicated. Replication approach would be good for analytics, if you need to analyze log data offline for calculating metrics or other batch related work, replication approach might be a good fit.

Efficient Data Transport

If you have multiple hosts running then logs data can accumulate quickly. There should be an efficient and reliable way to transport this data to the centralized application and ensure data is not lost. There are many frameworks available to transport log data. One way is directly plug input sources and framework can start collecting logs and another way is to send log data via API; application code is written to log directly to these sources it reduces latency and improves reliability.

Log Storage Solutions

Now we have transport in place, logs will need a destination, a storage where all the log data will be saved. The system should be highly scalable as the data will keep on growing and it should be able to handle the growth over time. Logs data will depend on how huge your applications are if your application is running on multiple servers or in many containers it will generate more logs.

Analysis and Alerting Mechanisms

Logs are meant for analysis and analytics. Once your logs are stored in a centralized location, you need a way to analyze them. There are many tools available for log analysis; if you need a UI for analysis, you can parse all the data in ElasticSearch and use Kibana or Greylog to query and inspect the data. Grafana and Kibana can be used to show real-time data analytics.

This is the last component in the centralized logging application. It’s nice to have an alerting system which will alert us to any change in the log patterns or calculated metrics. Logs are very useful for troubleshooting errors. It’s far better to have some alerting built into the logging application system which will send an email or notify us than to have someone keep watching logs for any changes.