Scaling with Prometheus: Managing 80M Metrics Smoothly

At Flipkart, we faced a challenge at the API Gateway, where roughly 2,000 instances of the gateway application were emitting close to 40,000 metrics each, resulting in over 80 million time-series datapoints being pushed at once. The metrics ranged from requests per second, diverse error rates, custom JMX metrics and detailed latency metrics such as p99, p95.

While StatsD worked at first, querying more than 3 hours of data became painful because it didn’t scale well. It choked the storage and made it practically impossible to query for long-term insights.

This article walks through how moving to prometheus helped us in this situation and also explains the architecture of a federated prometheus setup.

Some history: Gaps in existing tools

Before Prometheus, earlier monitoring tools had their own limitations:

• Push-based model: Tools like StatsD and Collectd, often combined with InfluxDB, relied on a push-based model for metric ingestion.

Press enter or click to view image in full size

• They relied on a weaker query language: Complex queries, especially multi-dimensional aggregations are more cumbersome in InfluxQL.

Example of a PromQL query without an InfluxQL equivalent:

• Ecosystem and Extensibility: Prometheus shines with first-class support for Kubernetes, Consul, AWS EC2, GCE, Azure, and file-based service discovery (full list here). It also has a vibrant ecosystem of exporters.
• Scaling Limitations: Prometheus supports hierarchical federation, making it easier to scale via metric filtering and enabling tiered aggregations. Most of the time a single instance is enough to able to handle a large number of metrics.

Architecture Components & Data Flow

• Service discovery & Exporters: Kubernetes pods (targets) expose metrics on an endpoint. Prometheus pulls from these endpoints.
• Prometheus: One Prometheus server is responsible for scraping and aggregating metrics. In our setup, this prometheus server drops instance label and exposes aggregated metrics over a new endpoint /federate, a federated prometheus server or job is responsible for scraping selected aggregated metrics and writes them to long-term storage
• Remote Storage & Grafana Dashboard: Long retention and visualization.

Dialing Down on Cardinality

• Drop instance label for metrics with the help of recording rules, keep stable labels like service & cluster.
• For counters (e.g., error count) and rate‑derived metrics (QPS/RPS/error‑rate), summing across instances gives the total while dropping the instance dimension.
• For latency (p95/p99), you can also publish avg/max/min across instances to capture typical vs. worst‑case — again without the instance label.
• If a service has N instances × M metrics in one cluster, dropping instance label collapses series by N times. E.g., 2,000 pods × 40k metrics → after rollups, ~40k series per cluster.

When Federation Helps (and When It Doesn’t)

Use it when:

• You have multiple clusters/regions and need org‑wide dashboards.
• To achieve a scalable Prometheus monitoring setup when a single server hits its CPU or memory.

Skip it when:

• You need metrics for per‑instance debugging globally.
• You have one small cluster where a single prometheus + storage would suffice.
• The plan is to mirror all raw metrics upward

Implementation Details

More to read:

• Federation
• Scrape Job Configs
• Recording Rules
• Mimir & Thanos