Datadog Reveals How to Measure Developer Experience in the AI Era
AI coding assistants are changing everything, and breaking old measurement models along the way. As these tools drive up pull request counts, commit frequency, and lines of code, individual output metrics are proving unreliable. Higher volume does not mean better productivity.
Research supports this. GitClear analyzed over 200 million lines of code and found that code churn nearly doubled after widespread AI adoption. More output, in other words, does not equal more stability.
That is why measuring developer experience in the AI era has become critical for engineering teams. Developer experience, commonly called DevEx, describes how an organization’s systems, workflows, tools, and culture affect developer productivity. It reflects what developers actually live through each day.
Datadog, which manages over 3,000 engineers in an AI-augmented software delivery lifecycle (SDLC), published a detailed breakdown of how it tracks DevEx across multiple dimensions. The approach combines system-level metrics with developer sentiment surveys, and the results are revealing.
The DevEx framework centers on three key dimensions: feedback loops, cognitive load, and flow state. Feedback loops cover build times, test results, and code review turnaround. Cognitive load tracks the mental effort needed to complete tasks. Flow state measures uninterrupted, energized focus.
In 2025, Datadog added a fourth dimension internally: AI adoption and impact. This measures how frequently engineers use AI coding tools and how those tools affect each stage of the SDLC.
To assess productivity, Datadog treats DORA metrics as a north star while also tracking supporting metrics across three categories: process efficiency, tool quality, and cognitive load.
Process efficiency metrics include time to PR ready, review time, merge time, and rollback-to-hotfix ratio. In an AI-augmented SDLC, PR throughput, the rate of merges across teams, becomes especially important. At Datadog, roughly 80% of PRs are now AI-assisted. Interestingly, AI does not significantly speed up individual changes. Instead, it enables developers to work on more changes simultaneously.
Tool quality metrics measure build and test duration, CI queue time, flaky test rate, and code coverage. Slow pipelines are a major source of developer friction. As AI accelerates code generation, the gap between producing changes and receiving build feedback becomes a serious bottleneck. Datadog addressed this by introducing persistent runners and improving CI speed by 50%.
Cognitive load is harder to measure directly, but proxies help. These include multi-agent orchestration complexity, discovery friction, environment parity issues, and unplanned work ratio. Datadog’s latest Engineering Experience survey found that incident-related toil has the strongest correlation with overall developer sentiment. To reduce discovery friction, Datadog uses its Internal Developer Portal (IDP) and Software Catalog, which automatically maintains up-to-date service ownership records. The Datadog MCP Server further cuts friction by giving AI agents direct access to live telemetry, logs, and runbook context.
Sentiment surveys complement all of this metric tracking. Datadog runs its Engineering Experience survey biannually. In the latest edition, engineers submitted over 2,400 free-text comments, exposing bottlenecks that metrics alone would have missed. Segmenting results by team also proved essential. Some teams saw review time increases of over 500%, even though the global average stayed stable.
Transparency matters too. Datadog follows a clear pattern when sharing results: “You said X, we shipped Y, metric Z improved.” Engineers need to know their feedback drives real change.
For teams starting from scratch, Datadog points to DORA Metrics, AI Impact, CI Visibility, Test Optimization, and the Datadog IDP as the core building blocks for measuring developer experience in the AI era.
