Agentic AI Slashes Spark Compute Costs by 44% at Datadog
Running Apache Spark at scale is expensive. Debugging it can take even longer. Datadog’s own engineering team faced both problems head-on, and their solution cut costs nearly in half.
The team’s ServiceQueryEdge job runs daily across seven datacenters, with individual partitions processing up to 27 TB of input and 16 billion records. Before optimization, it averaged $1,500 in infrastructure costs daily, with each run taking over 17 hours. That kind of overhead adds up fast. At those numbers, annual costs in just one datacenter approached $600,000.
To tackle it, the team paired Datadog’s Jobs Monitoring with an agentic AI approach built on Claude. The goal was not to let AI run free, it was to close the gap between what the Spark SQL execution plan reveals and what the source code explains. Correlating a slow operator in the SQL Plan back to the relevant section of application code can take hours, particularly for a large, complex job. The agent handled that correlation work automatically.
However, the team quickly learned that feeding an AI agent raw data at scale is its own problem. The agent exhausted its context window before completing meaningful analysis. Suggestions became incomplete or incoherent. The fix was to use subagents that retrieved only targeted, scoped data, preserving context for the analysis that actually mattered.
Even then, early recommendations missed the mark. Many addressed symptoms rather than root causes. So the team added a second subagent to act as a validator. Rather than generating new suggestions, it graded existing ones. It checked whether fixes actually addressed measured bottlenecks, whether Spark was already handling the issue automatically, and whether a proposed change could introduce new downstream problems.
The result was three high-impact optimizations. First, the team removed a redundant distinct() call that triggered unnecessary HashAggregate operations across every downstream stage. This reduced the number of rows being sent to subsequent stages to less than a tenth of the original size on some shards.
Second, they replaced a SortMergeJoin, which was forcing a sort of a multi-billion-row table to filter against a list of just 500 rows, with a broadcast join. The initial gains appeared modest. But once the third fix landed, the broadcast join’s full impact became clear.
That third fix addressed data skew. The salting logic created salts by applying modulus to a skewed column, meaning the resulting salt values inherited the same non-uniform distribution. Switching to rand() distributed rows evenly across partitions. Duration skew on the worst task dropped from a max-p50 of 25 minutes to around 10 minutes. Overall run duration fell by more than 57%.
Combined, the three changes delivered a 44% reduction in daily compute costs, from roughly $1,500 to $830 per day. In the US1 datacenter, run duration fell 60% and executor time dropped nearly 50%. At a pre-optimization annual cost of approximately $600k in US1, these gains represent an estimated $250k in annual savings from that datacenter alone. Across all other datacenters, the team projects an additional $50,000 in annual savings.
The agentic AI Spark cost optimization approach is now available to other teams through the Datadog MCP Server. Two purpose-scoped tools power it: get_spark_health for ranked stage-level bottlenecks, and get_spark_sql_plan for deep execution plan analysis. The MCP Server is generally available, and the data-observability toolset, including the Spark tools, is in Public Preview. Setup documentation is available on the Datadog docs site.
The team’s conclusion was direct: treat AI agents as collaborative partners, not autonomous solvers. The agent surfaced connections. The engineers made the calls. Together, they got results neither would have reached alone.
