Log Aggregation

In the context of state-of-the-art system monitoring, a long-established mantra states that both the metrics data and the log messages must be part of the dataset to be monitored, which only makes sense: The data gives you detailed information on errors that occur. The crux of the matter is that logs are still not standardized and therefore not machine-readable.

The field of log aggregation has been hotly contested on the market for more than 10 years. Splunk [5] is considered to be the king of the hill. No other solution is capable of reading and indexing log messages as comprehensively as this product. That said, the vendor extracts a high price for all this functionality. Although you can find alternatives such as the ELK stack (Elasticsearch, Logstash, Kibana) or Grafana Loki, they are not yet very widespread.

Netdata chooses a different approach: For more than a year, the manufacturer has quietly added various log aggregation functions to its tool. The messages become part of the metrics data that the agent sends to Netdata Cloud. The service evaluates these messages immediately and makes them part of regular monitoring. Anyone who uses Netdata's log aggregation feature therefore has access to the same functions for logs as for the metrics data itself, which makes it possible to search for specific details and alert on the basis of specific log entries.

Netdata and systemd integration is of central importance, of course. Netdata continuously reads the systemd journal, which is available on practically every recent Linux system, and forwards the details to Netdata Cloud. A file agent can also be used to collect the contents of text files. Netdata also automatically collects the logs from various services that are monitored anyway, unless you disable it. Once the log aggregation function has been activated, and because Netdata is monitoring MariaDB anyway, the agent also transmits the matching logs, which keeps the configuration overhead to a minimum.

Comprehensive Alerting

The best monitoring tools are useless if problems do not trigger alerts, and Netdata leaves nothing to chance. It comes with a pre-configured alerting setup for practically every collector (i.e., for the modules that aggregate the metrics data of a specific service) according to the various manufacturers' best practices and standards. As an admin, it is then up to you to configure the Netdata Cloud alerting module so that you receive the alerts.

The text messages that used to be commonplace are no longer the norm in increasing numbers of organizations. Instead, the focus is on cloud services and Netdata's capabilities are again quite impressive. It can contact the major US services (e.g., Opsgenie and PagerDuty, but also Discord, Mattermost, Rocket.Chat, Splunk, and Telegram). If you do want to send a short message, you can use a webhook and connect it to your SMS provider's API. If you don't want to use a centralized cloud setup for alerting, you can tell the Netdata agent to send the alerts directly. Even more contact options than with the cloud version are available, including good old IRC, email, and short message.

All told, Netdata helps you implement comprehensive alerting quickly, not least because the service's default settings are meaningful. In some situations, Netdata's developers have weighed up whether or not an alert is necessary (e.g., if one of dozens of web servers in a web server setup fails). The default configuration avoids an alert being sent out in the middle of the night, although you can adjust this setting if so desired.

Conclusions

Two things stand out when working with Netdata: The program launches incredibly quickly and works extremely well without further need for configuration. Without reservation, the developers keep the promise that virtually no other solution can produce usable results so quickly.

One drawback is that Netdata has to be operated as a cloud service with proprietary components, although you can run Netdata Cloud on your own infrastructure. The $10,000 or so that an on-premises setup on this scale costs [6] is probably not an issue for most organizations.

More serious is that every organization that uses Netdata has to accept vendor lock-in for part of its infrastructure. If this condition does not worry you, you will want to put Netdata on your shopping list, especially if you are looking to replace or at least upgrade an outmoded monitoring system.

The Author

Martin Loschwitz is the founder and managing director of True West IT Services GmbH, which offers scalable IT infrastructure based on OpenStack and Kubernetes.