# AX Check: datadog.com Checked 2026-10-06. Datadog setup and pricing are clear and ready to use. 21 of 23 checklist items passed: install steps, code samples, API key prereqs, and pricing all shown plainly with no login wall. ## Onboarding needs a login ## Coding sessions All three independent sessions (DeepSeek V4.1 Flash, Kimi K3, Qwen 3.8 Max) completed the task and produced per-unit pricing tables (per host, per GB, per metric), each noting assumptions like plan tier, free-tier limits, and regional list-price differences. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/ced6a06f-0cb1-4a5e-995a-17a3083f97ca) · [Read transcript](https://www.ax-check.com/datadog.com/sessions/deepseek.json) Final output gives per-SKU pricing (hosts, APM tiers, logs, custom metrics) and names assumptions: 'Numbers are ballpark; verify on their pricing page', free tier caps (5 hosts, 1-day retention), SaaS-only. #### End-to-end onboarding - **Onboarding needs a login**: Agent checked environment for Datadog credentials, found none (DD_API_KEY/DD_APP_KEY unset), and could not self-serve new ones. It built the example script and README, then tested with a deliberately fake key, getting HTTP 403 Forbidden - this proves the request plumbing works but is not an authenticated success. No real credentials were ever obtained or used, so no authenticated product operation occurred. Event 14: ```text But there are **no Datadog credentials** in this environment (`DD_API_KEY`/`DD_APP_KEY` unset), so I can build and run the request path but can't successfully write data. ``` Event 26: ```text HTTP 403 from GET /api/v1/validate { "errors": [ "Forbidden" ] } validate -> HTTP 403: {"errors": ["Forbidden"]} exit=1 ``` Event 28: ```text **I'm blocked on credentials.** There is no `DD_API_KEY`/`DD_APP_KEY` in this environment, so I can't perform a successful write or read. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Datadog API/App keys available in sandbox**: The sandbox environment had no DD_API_KEY or DD_APP_KEY set, and the agent had no self-service way to generate Datadog credentials (this requires a human to sign up and create keys in the Datadog org settings). This is a normal credential/login requirement, not a product defect — the agent correctly identified it, built the full request path anyway, and tested it with a dummy key to prove the plumbing worked before stopping as instructed. Event 4: ```text env | grep -iE 'datadog|dd_|api_key|app_key' ``` Event 14: ```text there are **no Datadog credentials** in this environment (`DD_API_KEY`/`DD_APP_KEY` unset), so I can build and run the request path but can't successfully write data ``` Event 28: ```text I'm blocked on credentials.** There is no `DD_API_KEY`/`DD_APP_KEY` in this environment, so I can't perform a successful write or read. ``` #### Suggested Changes None identified in this transcript. ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/9e8798e9-13da-4b0f-a9a1-c21c50e2d08f) · [Read transcript](https://www.ax-check.com/datadog.com/sessions/kimi.json) Final output gives per-host/per-GB/per-metric figures and explicitly ties them to assumptions (Pro/Enterprise plan tier, free tier of 5 hosts, 14-day trial, per-dimension usage). #### End-to-end onboarding - **Onboarding needs a login**: Agent never obtained Datadog credentials on its own. It wrote a script requiring DD_API_KEY and DD_APP_KEY, checked the environment, found none, and halted. No self-service signup or key generation was attempted or possible within the sandbox; the agent explicitly told the user to supply credentials from a free trial account. Event 19: ```text Missing credentials. Set DD_API_KEY and DD_APP_KEY env vars first. Command exited with code 1 ``` Event 21: ```text Datadog has no sandbox/anonymous mode: every API call needs an `DD_API_KEY` + `DD_APP_KEY` from a Datadog org. There are no credentials in this environment, so I can't execute the script or verify the responses. ``` Event 14: ```text if not os.environ.get("DD_API_KEY") or not os.environ.get("DD_APP_KEY"): raise SystemExit( "Missing credentials. Set DD_API_KEY and DD_APP_KEY env vars first." ) ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Datadog API/App keys available in sandbox**: Datadog requires an API key and Application key for any authenticated call; there is no anonymous or sandbox mode. The test environment had no credentials pre-provisioned and the agent has no way to self-register a Datadog account, so the example script could not be executed or verified. This is a missing-credentials limitation of the test setup, not a product defect. Event 19: ```text Missing credentials. Set DD_API_KEY and DD_APP_KEY env vars first. Command exited with code 1 ``` Event 21: ```text Datadog has no sandbox/anonymous mode: every API call needs an `DD_API_KEY` + `DD_APP_KEY` from a Datadog org. ``` #### Suggested Changes None identified in this transcript. ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/04113f0d-5243-45b5-9c55-c3569a0a0e36) · [Read transcript](https://www.ax-check.com/datadog.com/sessions/qwen.json) Final output pricing table is tied to explicit assumptions: 'US list prices; EU/US3/US5/Gov differ, and real contracts are negotiated', annual vs on-demand rates, and per-unit basis (host/GB/events). #### End-to-end onboarding - **Onboarding needs a login**: No real Datadog credentials existed in the sandbox (only an unrelated gateway key). The agent built a quickstart and ran it in mock mode (fake in-process transport, no real API) and live mode with self-typed bogus placeholder keys, which reached the real API but returned 401/403. No genuine API key was ever obtained or used, so no authenticated product operation succeeded against the hosted product. Event 13: ```text ls: cannot access '/sandbox/.datadog*': No such file or directory ``` Event 222: ```text [1] submit failed: ForbiddenException (HTTP 403) -> DD_API_KEY is invalid, revoked, or lacks the 'metrics_write' scope. Regenerate it in Org Settings -> API Keys. ``` Event 200: ```text BLOCKED: no Datadog credentials found. export DD_API_KEY= # Org Settings -> API Keys export DD_APP_KEY= # Org Settings -> Application Keys ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Datadog API/app keys available in the sandbox**: The environment has no Datadog credentials (checked env vars, dotfiles, config dirs — only an unrelated internal gateway key was present). This is a test-environment/credential limitation, not a product defect: Datadog requires normal account signup and key generation, which is expected behavior. The agent correctly flagged this rather than faking success, and confirmed via a live run with placeholder keys that the real API responds with 401/403 as expected once credentials are missing or invalid. Event 13: ```text ls: cannot access '/sandbox/.datadog*': No such file or directory ``` Event 222: ```text [2] estimate skipped: UnauthorizedException: (401) Reason: Unauthorized ``` - **Datadog pricing page is a client-side rendered SPA, hard to scrape**: The public pricing page returned identical byte content for every product-specific URL query parameter, forcing the agent to parse embedded script/HTML text with regex instead of a clean data source. This is product-site behavior (not an agent error), and it only slowed down, did not block, the price research, since the agent successfully extracted numbers after extra effort. Event 28: ```text infrastructure: 1638768 bytes log-management: 1638768 bytes apm: 1638768 bytes ``` #### Suggested Changes - **Fix the mislabeled Universal Service Monitoring price card on the pricing page**: On datadoghq.com/pricing, the $9/$13 per-host 'Starting At' price card sits in a layout position that is easy to misattribute to Cloud SIEM; the agent initially mis-tagged it as Cloud SIEM pricing before catching the error via wider-context scraping. Check the pricing page's visual/DOM grouping so the Universal Service Monitoring card is unambiguously separated from the Cloud SIEM section above it. Event 244: ```text Universal Service Monitoring Starting At $ 9 Per Infrastructure Monitoring host, per month* ... **Billed annually or $ 13 on-demand ``` Event 246: ```text Important correction found: the `$9/$13` item is **Universal Service Monitoring**, not Cloud SIEM. Let me find the real Cloud SIEM price. ``` - **Document the exact metric-estimate endpoint path and response shape**: The estimate_metrics_output_series SDK method actually calls GET /api/v2/metrics/{metric_name}/estimate and returns estimated_output_series as a single int, but nothing discoverable without reading SDK source signaled this; the agent's first guess (different path, list response) was wrong and required source inspection to fix. Add this endpoint path and response field type to the datadog-api-client method docstring or online API reference for MetricsApi.estimate_metrics_output_series. Event 170: ```text GET https://api.datadoghq.com/api/v2/metrics/system.cpu.user/estimate ``` Event 172: ```text Two real bugs found: the endpoint is `/api/v2/metrics/{name}/estimate`, and `estimated_output_series` is an **int**, not a list. ``` - **Document the filter_hours_ago minimum value for the metrics estimate endpoint**: Calling estimate_metrics_output_series with a small filter_hours_ago (e.g. 3) throws a client-side validation error because the real minimum is 49 hours, which is not mentioned anywhere the agent could see except the generated SDK source's validation dict. Add this constraint to the public API reference for the filter[hours_ago] query parameter on the metrics estimate endpoint. Event 135: ```text [2] estimate skipped: ApiValueError: Invalid value for `filter_hours_ago`, must be a value greater than or equal to `49` ``` ### Task given to each agent Help me build a simple example using Datadog. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: B · 84/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a text/markdown request; no Markdown representation offered. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs, products, knowledge center, blog, and support resources with descriptions. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt organizes links under clear sections like Core Products, Solutions, and Support Resources. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt links MCP Server/CLI page and API docs; skills documented in docs. ``` - **Pass** — A compact guide representation exists ```text llms.txt links to .md product pages; docs pages serve text/markdown representations. ``` - **Pass** — A focused guide is directly retrievable ```text Agent Observability quickstart .md fetched directly with install and run steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text Quickstart markdown is 2646 tokens, well under the 8000-token budget. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be judged. ``` - **Pass** — The compact guide is independently actionable ```text Agent Observability quickstart gives concrete install and run steps for Python, Node.js, and Java. ``` - **Pass** — Install and next-step links resolve ```text Quickstart's next-step links resolve, e.g. evaluations and SDK docs fetched successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quickstart page gives concrete Python, Node.js, Java instrumentation steps and a Hello World app. ``` - **Pass** — Installation commands are extractable ```text Quickstart shows extractable install commands: pip install ddtrace, npm install dd-trace, wget dd-java-agent.jar. ``` - **Pass** — Code examples are available without interaction ```text Quickstart includes full Python and Node.js Hello World code examples inline, no interaction needed. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Prerequisites state a Datadog API key is required and link where to find it. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders full plan cards with prices in static HTML, no interaction needed. ``` - **Pass** — Prices are stated, not gated ```text Concrete prices shown: Infrastructure $15/host, Logs $0.10/GB, AI Credits $500. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: per host, per GB, per million events, per committer, per credit. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Final output gives per-SKU pricing (hosts, APM tiers, logs, custom metrics) and names assumptions: 'Numbers are ballpark; verify on their pricing page', free tier caps (5 hosts, 1-day retention), SaaS-only. Kimi K3: Final output gives per-host/per-GB/per-metric figures and explicitly ties them to assumptions (Pro/Enterprise plan tier, free tier of 5 hosts, 14-day trial, per-dimension usage). Qwen 3.8 Max: Final output pricing table is tied to explicit assumptions: 'US list prices; EU/US3/US5/Gov differ, and real contracts are negotiated', annual vs on-demand rates, and per-unit basis (host/GB/events). This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Datadog API Reference page documents the HTTP REST API with auth and client libraries. ``` - **Pass** — An MCP server is documented and well-formed ```text MCP Server docs cover setup, toolsets, auth, permissions, and supported clients. ``` - **Pass** — A CLI install path is documented ```text Pup CLI page documents Homebrew, source build, and manual download install paths. ``` - **Pass** — SDK packages resolve on their registries ```text Registry lookups for pypi datadog, datadog-api-client, ddtrace and npm dd-trace all returned HTTP 200. ``` - **Pass** — Agent skills are published ```text Agent skills published: dd-software-delivery skills repo and Pup's pup skills install. ``` [Full report data](https://www.ax-check.com/datadog.com/report.json)