Scaling Observability Across Massive Traffic Spikes
DoorDash connects hundreds of millions of consumers with local merchants worldwide, where software reliability directly impacts merchant revenue and consumer trust. High-demand events like Valentine’s Day and the Super Bowl underscored the need for observability infrastructure that could scale efficiently with DoorDash’s traffic and telemetry volumes.
Additionally, as the business grew and DoorDash expanded its microservices architecture, managing high cardinality and escalating telemetry costs across thousands of services became a key engineering priority. DoorDash aimed to make reliability goals consistently measurable across teams, giving business and technical leaders a shared view of service health and overall system reliability.
Automated SLO Governance and Control Plane Efficiency
DoorDash equipped engineers with turnkey SLO automation built directly into service code for ease of use, establishing a tiered reliability framework across all applications. These 70,000 SLOs carry over to the Cortex XCOR platform for monitoring service health, creating a shared operational language that connects technical metrics to business health and global order fulfillment.
To control escalating observability costs, DoorDash utilized the Cortex XCOR Optimization Engine. Based on the solution’s automated recommendations, DoorDash’s Observability Team implemented over 325,000 recording rules to support service monitoring and used aggregation rules to achieve a 92% data optimization rate, reducing low-signal noise and telemetry overhead. Along with lowering costs, this approach equips the engineering team with critical real-time visibility to keep DoorDash services available and reliable.
"Palo Alto Networks has been a true thought partner as we designed our SLO system and tackled high-cardinality challenges. Automating SLO creation has given our engineers a consistent way to measure service health and keeps teams aligned on reliability."
— Steven Callister
Software Engineer, Reliability Platform Team, DoorDash