Photo by FLOUFFY on Unsplash
Long-Term Prometheus Data Storage with Cortex
Trend Scout
Prometheus suffers from a structural problem: It does not offer a true cluster mode. A single instance stores its data locally and responds to queries from this local database. High availability therefore requires a separate design. Many teams solve this problem with the use of two Prometheus instances that query the same targets and with graphical or logical abstractions that merge the two data sources (e.g., in Grafana).
This solution increases availability, but it does not eliminate the fundamental problem of scaling. Each instance continues to back up locally, each instance compresses its own data, and each instance only stores its own data. Metrics volumes that grow and retention times that become longer mean more than a significant loss of convenience, because you have to deal with multiple points of administration. If you have several locations with the same setup, the unpredictability of slow connections between them adds to the problem.
In these scenarios, Cortex [1] [2] enters the scene. Cortex is directly related to Prometheus, because it operates in the same data model and protocol world and natively understands Prometheus data. It fields Prometheus metrics, stores them long term on scalable back ends, and makes them available again for queries. Instead of each Prometheus instance keeping its entire dataset locally, Prometheus transfers the data to Cortex at defined intervals, typically with its remote write module. Cortex then assumes responsibility for long-term storage and distributes the data across multiple instances of itself that scale horizontally, which means you can offload the pressure from the individual Prometheus instance to a system designed for scaling.
Monitoring
Containers have secured their place in everyday IT, and they are here to stay. As new
...Buy this article as PDF
(incl. VAT)
Buy ADMIN Magazine
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Most Popular
Support Our Work
ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
