# AX Check: checklyhq.com Checked 2026-09-24. Checkly's docs and pricing are fully machine-readable, no gates. All 23 checklist items passed: quickstart, CLI, API, MCP and skills all resolve, and pricing tiers (Hobby $0, Starter $29, Team $80) are stated with clear limits. ## Onboarding needs a login ## Coding sessions All three independent agent sessions completed and found pricing directly on the pricing page, reporting a consistent tiered structure (Hobby free, Starter and Team paid tiers, custom Enterprise) along with usage assumptions like billing cadence and run limits. ### DeepSeek V4 Pro [View public run](https://agents.withgauge.com/p/runs/1b946db5-0965-41b6-a8d1-99f116d34863) · [Read transcript](https://www.ax-check.com/checklyhq.com/sessions/deepseek.json) Final output gives a pricing table (Hobby $0, Starter $24/mo, Team $64/mo, Enterprise custom) sourced from checklyhq.com/pricing/ (seq 14) with explicit assumptions: billed annually, run caps per plan, overage rates, and add-ons called out separately. #### End-to-end onboarding - **Onboarding needs a login**: The agent scaffolded a full Checkly Monitoring-as-Code project (config, API check, alert channel) using the npm package installed locally, and confirmed the code type-checks. It never obtained real Checkly credentials: no CHECKLY_API_KEY/CHECKLY_ACCOUNT_ID existed in the sandbox env, and `npx checkly login` requires an interactive browser which the agent could not complete. Both `checkly whoami` and `checkly test` failed with an explicit auth-required error, and the session ended there without any authenticated call against the real Checkly API succeeding. Event 8: ```text ALG_INFERENCE_MODEL= PI_MODEL= PI_GATEWAY_API_KEY= GIT_CONFIG_KEY_0= ``` Event 171: ```text Error: Run `npx checkly login` or set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` in your environment or .env file. ``` Event 9: ```text checkly.com -> HTTP 200 api.checklyhq.com -> HTTP 401 ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Checkly credentials available to authenticate the CLI**: The sandbox environment had no CHECKLY_API_KEY or CHECKLY_ACCOUNT_ID, and `npx checkly login` requires opening a browser for interactive OAuth, which isn't possible in this headless session. This is a normal login requirement for a credential-gated SaaS product, not a product defect. It stopped the agent from running `checkly test` or `checkly deploy` against the real account, even though the code itself was fully scaffolded and type-checked. Event 171: ```text Error: Run `npx checkly login` or set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` in your environment or .env file. ``` Event 9: ```text api.checklyhq.com -> HTTP 401 ``` #### Suggested Changes None identified in this transcript. ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/5cb982c7-9257-4a6b-bba2-e7458fe46276) · [Read transcript](https://www.ax-check.com/checklyhq.com/sessions/kimi.json) Final output gives tiered pricing (Hobby $0, Starter $24, Team $64, Enterprise custom) with named assumptions: monthly billing, annual discount ~20%, usage/plan limits, sourced from checklyhq.com/pricing (seq 73,77). #### End-to-end onboarding - **Onboarding needs a login**: Agent never obtained real Checkly credentials. It checked environment variables (found none), hit 401 on the public API, installed the CLI, and found that `checkly login` is browser-interactive only with no headless/API signup path. It explicitly stopped short of running `checkly test` or `checkly deploy` and said so in its final summary, so no authenticated product operation occurred. Event 16: ```text PI_GATEWAY_API_KEY=unused exit: 0 ``` Event 44: ```text Error: Run `npx checkly login` or manually set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` environment variables to setup authentication. ``` Event 84: ```text `checkly login` is browser-interactive only (no headless signup/API), and there are no `CHECKLY_API_KEY`/`CHECKLY_ACCOUNT_ID` credentials in this environment. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No headless login path for Checkly CLI**: `checkly login` requires an interactive browser flow and no CHECKLY_API_KEY/CHECKLY_ACCOUNT_ID were present in the sandbox, so `checkly test` and `checkly deploy` both failed with an authentication error. This is a normal product login requirement (not a defect) combined with the test environment lacking credentials, and it stopped the agent from running any check against Checkly's hosted infrastructure. Event 44: ```text Error: Run `npx checkly login` or manually set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` environment variables to setup authentication. ``` Event 58: ```text Error: Run `npx checkly login` or set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` in your environment or .env file. ``` #### Suggested Changes - **Surface a non-interactive/CI signup or API-key-only account creation path in CLI login help**: `npx checkly login --help` only describes the browser-based OAuth flow; there is no documented way to create an account and API key entirely from the terminal. Adding a documented CI-token or headless signup option (or clearly pointing to where to generate a CHECKLY_API_KEY without a browser) would let CLI-only environments get past the login wall. Verify by running `npx checkly login --help` and confirming a non-interactive option appears alongside the browser flow. Event 52: ```text Login to your Checkly account or create a new one. USAGE $ checkly login ``` Event 44: ```text Error: Run `npx checkly login` or manually set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` environment variables to setup authentication. ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/26e7f19e-49d7-4a4d-a06c-4d7fec6fd146) · [Read transcript](https://www.ax-check.com/checklyhq.com/sessions/qwen.json) Final summary cites live-fetched (seq 13) tiered pricing (Hobby $0, Starter $24, Team $64, Enterprise custom) with explicit assumptions: check frequency (5 min), 2 locations, resulting ~8.6k runs/mo estimate placed in the Hobby tier. #### End-to-end onboarding - **Onboarding needs a login**: Agent scaffolded a Checkly monitoring-as-code project (config + one ApiCheck) and confirmed the SDK constructs load, but never obtained real Checkly credentials. Running `npx checkly test` failed immediately because no account/API key existed, and the agent explicitly asked the human to supply CHECKLY_API_KEY and CHECKLY_ACCOUNT_ID to proceed. No authenticated operation against the hosted Checkly product occurred. Event 32: ```text Error: Run `npx checkly login` or set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` in your environment or .env file. ``` Event 36: ```text ApiCheck: function | Frequency.EVERY_5M: Frequency { frequency: 5, frequencyOffset: undefined } | AssertionBuilder: function ``` Event 38: ```text Give me those two env vars and I can finish the deploy in one step. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Checkly account credentials available in sandbox**: The sandbox had no CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID, or logged-in session. Both `npx checkly test` and the planned `npx checkly deploy` require hosted authentication, so the agent could only validate that SDK constructs load locally, not actually run or deploy a check against the live Checkly service. This is a missing-credentials limitation of the test environment, not a product defect — Checkly's CLI behaved exactly as documented by erroring out and telling the agent how to authenticate. Event 32: ```text Error: Run `npx checkly login` or set `CHECKLY_API_KEY` & `CHECKLY_ACCOUNT_ID` in your environment or .env file. ``` #### Suggested Changes None identified in this transcript. ### Task given to each agent Help me build a simple example using Checkly. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 100/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown with Vary: Accept when requested with Accept: text/markdown. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs, API reference, CLI, MCP, skills, OpenAPI, and guides with clear organization. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt gives 'When to use Checkly' guidance and 'How an agent should call Checkly' routing. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt details CLI, MCP endpoint/manifest, Agent Skills, and REST API/OpenAPI. ``` - **Pass** — A compact guide representation exists ```text Homepage returns text/markdown via content negotiation; hand-authored .md twins like /pricing.md and /cli.md exist. ``` - **Pass** — A focused guide is directly retrievable ```text Quickstart page fetched as markdown with concrete steps: npx checkly init, test, deploy. ``` - **Pass** — Equivalent instructions fit a token budget ```text Quickstart markdown is 2191 tokens, well under 8000 and below its HTML extraction. ``` - **Pass** — Product-docs links survive format changes ```text Markdown homepage retains docs, CLI, API, MCP and skills links; equivalent routes survive. ``` - **Pass** — The compact guide is independently actionable ```text Quickstart gives concrete steps: npx checkly init, login, write check, test, deploy, alerts. ``` - **Pass** — Install and next-step links resolve ```text Fetched quickstart, browser-check quickstart, CLI, API, MCP and skills pages all returned 200. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quickstart page gives concrete CLI steps: init, login, test, deploy. ``` - **Pass** — Installation commands are extractable ```text Install commands shown: npx checkly init, npm i --save-dev checkly. ``` - **Pass** — Code examples are available without interaction ```text Quickstart shows Playwright and alert-channel code examples inline without interaction. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Prerequisites list Node.js 20.19+/22.12+; login via npx checkly login authenticates. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders full plan tables and prices as static Markdown, no interaction needed. ``` - **Pass** — Prices are stated, not gated ```text All plan prices stated explicitly: Hobby $0, Starter $29, Team $80, Enterprise custom. ``` - **Pass** — Pricing units and limits are explicit ```text Units and limits explicit: monitors, browser/API runs, overage rates, locations, users. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4 Pro: Final output gives a pricing table (Hobby $0, Starter $24/mo, Team $64/mo, Enterprise custom) sourced from checklyhq.com/pricing/ (seq 14) with explicit assumptions: billed annually, run caps per plan, overage rates, and add-ons called out separately. Kimi K3: Final output gives tiered pricing (Hobby $0, Starter $24, Team $64, Enterprise custom) with named assumptions: monthly billing, annual discount ~20%, usage/plan limits, sourced from checklyhq.com/pricing (seq 73,77). Qwen 3.8 Max: Final summary cites live-fetched (seq 13) tiered pricing (Hobby $0, Starter $24, Team $64, Enterprise custom) with explicit assumptions: check frequency (5 min), 2 locations, resulting ~8.6k runs/mo estimate placed in the Hobby tier. This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text OpenAPI 3.0 spec served at /openapi.json and api.checklyhq.com/openapi.json; API reference docs reachable. ``` - **Pass** — An MCP server is documented and well-formed ```text MCP server documented with Streamable HTTP endpoint, manifest, server card, OAuth and API-key auth. ``` - **Pass** — A CLI install path is documented ```text CLI install documented: npm install -g checkly, npx checkly init, plus direct CDN downloads. ``` - **Pass** — SDK packages resolve on their registries ```text npm registry lookup for the checkly package returned HTTP 200. ``` - **Pass** — Agent skills are published ```text Agent skills published: npx checkly skills install, four skills documented, machine-readable index linked. ``` [Full report data](https://www.ax-check.com/checklyhq.com/report.json)