# AX Check: keploy.io Checked 2026-10-06. Docs and pricing are clear, but the quickstart is just a menu Install steps, CLI commands and pricing are all stated plainly, but the quickstart page itself is only a language chooser with no concrete first steps, and the documented MCP server link returns a 404. ## Onboarding needs a login ## Coding sessions All three independent sessions (DeepSeek V4.1 Flash, Kimi K3, Qwen 3.8 Max) completed their task and correctly reported pricing, citing Playground free tier, Pro at roughly $19-24 per user/month, and custom Enterprise pricing, each with explicit usage-cap assumptions drawn straight from the pricing page. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/3cdc8b5b-d025-4207-984f-5ad3d0e6486e) · [Read transcript](https://www.ax-check.com/keploy.io/sessions/deepseek.json) Final output gives a pricing table (Playground free w/ monthly quotas, Pro $19/user/mo + usage, Enterprise custom) sourced from keploy.io/pricing (seq 10), with named assumptions like plan tier, usage credits, and overage billing basis. #### End-to-end onboarding - **Onboarding needs a login**: Agent built a working hosted-API example script but never obtained a real Keploy PAT (kep_...). It tested with a fake key and got a genuine 401 from the live server, confirming the API works, but no authenticated write operation succeeded since minting a token requires human browser sign-in with no self-service bootstrap available in-session. Event 138: ```text I need a `kep_...` API key, and I can't mint one from here. Keploy's hosted API authenticates with a scoped Personal Access Token created in the browser console at `app.keploy.io/settings/api-keys` (or auto-provisioned by the VS Code extension). There's no keyless bootstrap for the authenticated endpoints. ``` Event 124: ```text Error: GET /apps -> HTTP 401 AUTHENTICATION_REQUIRED invalid API key ``` Event 86: ```text To use it, every tool call must include a Personal Access Token (PAT) in the `Authorization` header. Follow these steps: ## 1. Create a PAT - Sign in at https://app.keploy.io ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service way to obtain a Keploy API key**: The hosted REST/MCP API requires a scoped kep_ Personal Access Token, which can only be created via a human browser sign-in at app.keploy.io/settings/api-keys (or via the VS Code extension's auto-provisioning). The sandbox had no such credential and no automated way to generate one, so every authenticated call (list apps, create app, generate tests, run tests) was blocked. This is a normal product login requirement, not a defect, but it stopped the session from completing the actual end-to-end demo. Event 138: ```text I need a `kep_...` API key, and I can't mint one from here. ``` Event 86: ```text ## 1. Create a PAT - Sign in at https://app.keploy.io - Open **Settings → API Keys**: https://app.keploy.io/settings/api-keys ``` - **Several guessed documentation URLs returned 404**: The agent tried a few plausible quickstart paths that all 404'd before it found the correct paths via llms.txt. These look like reasonable guesses rather than confirmed hallucinations causing a stuck session; the agent recovered immediately by fetching the sitemap-like llms.txt file. Event 24: ```text === https://docs.keploy.io/quickstart 404 === https://docs.keploy.io/server/v2/quickstart 404 === https://docs.keploy.io/server/v3/quickstart 404 ``` Event 42: ```text 404 Not Found 404 Not Found Code: NoSuchKey Message: The specified key does not exist. Key: docs/running-keploy/public-rest-api/index.html ``` #### Suggested Changes - **Add a keyless or CLI-based bootstrap path for the first API key**: Document or build a way to mint the first kep_ PAT without a browser session (e.g., a CLI device-code flow) for agent/CI contexts that have no interactive login. Verify by having a headless script obtain a working key end to end without a human pasting a token into app.keploy.io/settings/api-keys. Event 86: ```text You need an existing `admin`-scoped key to create keys via the API. To bootstrap your first key, ask your organization admin or create one from the [Keploy Console](https://app.keploy.io). ``` - **Fix or redirect the broken docs.keploy.io/quickstart link**: The top-level /quickstart and /server/v2|v3/quickstart paths 404; the working path is /docs/quickstart/quickstart-filter/. Add redirects from the shorter guessed paths, and confirm with a direct curl returning 200. Event 24: ```text === https://docs.keploy.io/quickstart 404 ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/d29d679d-a319-4a61-92d3-0c782c99de4c) · [Read transcript](https://www.ax-check.com/keploy.io/sessions/kimi.json) Final output lists Playground (free, no credit card, 30 suite gens/mo, 100 test runs/mo, 5 AI credits), Pro ($19/user/mo + usage credit), and Enterprise (contact sales), with the explicit assumption that 'the demo fits comfortably in this tier' (Playground). #### End-to-end onboarding - **Onboarding needs a login**: Agent found and documented a genuine hosted REST API workflow (api.keploy.io/client/v1), confirmed it live with unauthenticated/bad-key probes that returned real error envelopes, and wrote a working script exercising the full create-app/generate/run/report flow. However, the agent never obtained its own API key. Signup requires human OAuth login, and the final script run with no key exits with an instruction to sign up manually. No authenticated operation against the real product was ever completed. Event 120: ```text KEPLOY_API_KEY is not set. Get a free key: sign up at https://app.keploy.io, then in the Console create an API key with read+write scopes, and: export KEPLOY_API_KEY=kep_... exit=1 ``` Event 123: ```text FAILED at auth check (GET /users/me): HTTP 401 AUTHENTICATION_REQUIRED invalid API key exit=1 ``` Event 130: ```text I can't create the Keploy account myself — signup at app.keploy.io requires human OAuth (GitHub/Google). To run the example: 1. Sign up free at https://app.keploy.io ``` #### Hallucinated URLs - **Guessed API testing docs paths returned 404**: Agent guessed several plausible-looking docs URLs (public-rest-api, quickstart, ai-test-generation under /docs/api-testing/) before finding the real path via the sitemap. These were invented guesses, not links encountered in any page, and all three failed with 404, so this is a clear case of invented-and-failed URLs rather than an ambiguous broken link. Event 38: ```text == https://keploy.io/docs/api-testing/public-rest-api/ -> 404 == https://keploy.io/docs/api-testing/quickstart/ -> 404 == https://keploy.io/docs/api-testing/ai-test-generation/ -> 404 ``` #### Blockers - **No self-service path to an API key**: Keploy account creation requires signing in via app.keploy.io (OAuth), which is normal human login, not a product defect. This is a session/environment limitation: the agent cannot complete OAuth itself, so the finished example script cannot be run end-to-end to a successful authenticated call within the session. Event 120: ```text KEPLOY_API_KEY is not set. Get a free key: sign up at https://app.keploy.io, then in the Console create an API key with read+write scopes ``` Event 130: ```text I can't create the Keploy account myself — signup at app.keploy.io requires human OAuth (GitHub/Google). ``` - **Hosted test-run API rejects localhost targets**: Product behavior, not an error: the OpenAPI spec explicitly documents that /apps/{appId}/test-suites/run only accepts public base URLs and rejects loopback/private IPs with a 400, pushing local-app testing onto a separate local agent/CLI workflow that this session was told to avoid. Event 93: ```text Run test suites against a PUBLIC target URL. DO NOT use for local-app / localhost runs — base_url must be reachable from the SaaS backend (rejects loopback / private IPs as 400 'invalid baseURL'). ``` #### Suggested Changes - **Add a direct link from the pricing page to API key setup**: On https://keploy.io/pricing, the Playground plan is described but there's no link to the Console API Keys page or the Public REST API quickstart; the agent had to independently find /docs/running-keploy/public-api/ via sitemap crawling after three guessed URLs 404'd. Add a 'Get your API key' link on the pricing/signup page pointing to the real API docs path, and verify it by checking the link resolves to a 200 page describing key creation. Event 32: ```text Trying AI test generation Playground The perfect starting place to try Keploy and generate your first tests. Free forever. ``` Event 38: ```text == https://keploy.io/docs/api-testing/public-rest-api/ -> 404 ``` - **Fix broken /docs/api-testing paths or add redirects**: Several plausible docs paths under /docs/api-testing/ returned 404 even though a Public REST API doc exists at a different path (/docs/running-keploy/public-api/). Add redirects from the more intuitive /docs/api-testing/* paths to the actual public-api doc location, and confirm by curling the old paths and checking for a 200 or 301 instead of 404. Event 38: ```text == https://keploy.io/docs/api-testing/quickstart/ -> 404 == https://keploy.io/docs/api-testing/ai-test-generation/ -> 404 ``` Event 65: ```text Public REST API | Keploy Docs Skip to main content Products Integration Testing API Testing API Reference Public REST API ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/087e233d-cb8c-45b3-9bfb-d0a57e4bd2e1) · [Read transcript](https://www.ax-check.com/keploy.io/sessions/qwen.json) Final output (seq 27) lists Open Source/Playground/Pro ($24-19/user/mo)/Enterprise tiers with explicit assumptions (seat-based, monthly vs annual, usage caps, overage rates). #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: The agent never attempted to sign up, obtain an API key/token, or call any hosted Keploy endpoint. It only fetched public marketing/docs/pricing pages with curl and then stopped, concluding the product requires a local service stack. No credential acquisition or authenticated operation occurred. Event 27: ```text the quickstart requires a local service stack and long-running processes — I'm saying so plainly and stopping. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No hosted SDK/API path found for a lightweight example**: Agent behavior/analysis, not a tool failure: after reading the docs and pricing pages, the agent determined Keploy's core workflow requires running the user's own application locally under its recording proxy (eBPF/user-space interception) to capture traffic, then replaying it — there is no standalone hosted API to build a simple example against without a local service. This matches the seed prompt's instruction to stop if the quickstart requires a local stack or long-running commands, so the agent halted rather than proceeding. Event 27: ```text Keploy has no hosted SDK/API workflow I can act on for a "simple example." Its entire product model is: ``` Event 27: ```text The hosted cloud product (`app.keploy.io`) still depends on that locally-recorded traffic — it's a dashboard/AI layer over captures, not a standalone API you can build against. ``` #### Suggested Changes - **Add a hosted-only quickstart that doesn't require local traffic capture**: On the docs introduction/quickstart page (https://keploy.io/docs/keploy-explained/introduction/), add a path for a minimal example using only the hosted app.keploy.io API/SDK without needing eBPF or a locally running application. Verify by having a new user complete a working example purely through hosted API calls, with no local proxy or recording step required. Event 27: ```text Its "Public REST API" manages recorded artifacts, it can't produce a working example without a running local service stack. ``` ### Task given to each agent Help me build a simple example using Keploy. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: B · 77/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a text/markdown request; no Markdown representation offered. ``` - **Failed** — The compact guide is independently actionable ```text Quickstart page is only a language chooser; no concrete install or first-run steps. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs hub, quickstart, installation, CLI, API reference and MCP server. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt provides primary navigation plus a full annotated URL index of docs and pages. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt names the MCP server and links its docs; API reference is indexed. ``` - **Pass** — A compact guide representation exists ```text Docs pages offer 'View as Markdown'; quickstart-filter.md returns text/markdown. ``` - **Pass** — A focused guide is directly retrievable ```text quickstart-filter.md is a directly retrievable Markdown guide, though it is a chooser. ``` - **Pass** — Equivalent instructions fit a token budget ```text quickstart-filter.md is 103 tokens, far under the 8000-token budget. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be judged. ``` - **Pass** — Install and next-step links resolve ```text Fetched install, code-coverage and API-reference doc pages all returned HTTP 200. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Quickstart filter page offers language/environment chooser linking to per-language quickstarts. ``` - **Pass** — Installation commands are extractable ```text Install page gives curl install command plus Docker, Lima, WSL and manual binary steps. ``` - **Pass** — Code examples are available without interaction ```text Install and CLI pages show record/test commands inline without interaction. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Install page states free account needed and documents API key creation and login. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders plan cards and full comparison table without interaction. ``` - **Pass** — Prices are stated, not gated ```text Playground free forever; Pro $19/user/mo; Enterprise contact sales, all stated publicly. ``` - **Pass** — Pricing units and limits are explicit ```text Limits explicit: 30 test suites/mo, 100 test runs/mo, 5,000 sandbox runs, overage rates. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Final output gives a pricing table (Playground free w/ monthly quotas, Pro $19/user/mo + usage, Enterprise custom) sourced from keploy.io/pricing (seq 10), with named assumptions like plan tier, usage credits, and overage billing basis. Kimi K3: Final output lists Playground (free, no credit card, 30 suite gens/mo, 100 test runs/mo, 5 AI credits), Pro ($19/user/mo + usage credit), and Enterprise (contact sales), with the explicit assumption that 'the demo fits comfortably in this tier' (Playground). Qwen 3.8 Max: Final output (seq 27) lists Open Source/Playground/Pro ($24-19/user/mo)/Enterprise tiers with explicit assumptions (seat-based, monthly vs annual, usage caps, overage rates). This behavioural item does not affect the fast grade. ``` ### Activation - **Failed** — An MCP server is documented and well-formed ```text llms.txt advertises an MCP server at /docs/keploy-cloud/mcp, but that page returns 404. ``` - **Pass** — An API reference or OpenAPI spec is reachable ```text Public REST API reference documents base URL, auth, endpoints, and links to hosted API docs. ``` - **Pass** — A CLI install path is documented ```text Installation docs give concrete CLI install commands (curl install.sh, Homebrew, PowerShell) plus record/test usage. ``` - **Skipped** — SDK packages resolve on their registries ```text No SDK package registry lookup result was supplied in the fetched evidence. ``` - **Skipped** — Agent skills are published ```text No agent skills documentation or skills package was found in the fetched pages. ``` [Full report data](https://www.ax-check.com/keploy.io/report.json)