# AX Check: ploy.ai Checked 2026-10-21. Docs and pricing are clear, but agents don't always agree on price. 22 of 23 checklist items passed. Two sessions quoted exact tiers ($0/$50/$300/Custom) from the pricing page; one session found that page 404ing and reported no public pricing. ## Onboarding needs a login ## Coding sessions All three independent sessions completed. DeepSeek V4.1 Flash and Kimi K3 both read the pricing page and cited the same Free/$50/$300/Custom tiers with credit details. Qwen 3.8 Max instead hit a 404 on the pricing page and concluded the product was quote-based with no public price shown. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/ab05202a-9602-46a8-a328-980408052f48) · [Read transcript](https://www.ax-check.com/ploy.ai/sessions/deepseek.json) Gave Free/$50/$300/Custom tiers with credit allotments. Explained credits are spent only on active builds, plus the enrichment caps and domain-family rules. README wrongly puts some Starter features under Free. (seq 41, seq 86, README.md) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: The agent only exercised Ploy's unauthenticated public 'migration-estimate' marketing API (a lead-gen tool, not an authenticated product operation). It never obtained real Ploy account credentials, never ran 'ploy login', and never created/edited any resource (site, page, database) inside a Ploy workspace. It explicitly called out the gap itself rather than attempting self-service signup. Event 84: ```text Site: https://ploy.ai/ (scan: complete) Pages: 425 Platform: custom ``` Event 86: ```text the **authenticated** CLI/MCP flows (creating/publishing sites, reading visitors) require a Ploy account, and I don't have credentials in this environment, so I didn't exercise them ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No account credentials for authenticated CLI/MCP flows**: The CLI requires 'ploy login' and the MCP requires OAuth sign-in to do any real product work (publish sites, manage databases, read visitors). The sandbox had no Ploy credentials and no interactive browser, so the agent correctly stayed on the unauthenticated public API rather than attempting a login it couldn't complete. This is a session/environment limitation, not a product defect. Event 86: ```text the **authenticated** CLI/MCP flows (creating/publishing sites, reading visitors) require a Ploy account, and I don't have credentials in this environment, so I didn't exercise them ``` Event 66: ```text href="https://docs.ploy.ai/cli/install" ``` #### Suggested Changes - **Publish a lightweight unauthenticated SDK or sample alongside the CLI/MCP docs**: The only hosted, no-auth developer endpoint found (ploy.ai/agent-tools/migration-estimate) is a marketing lead-gen tool, not a general-purpose API for building on Ploy; no npm package named ploy/sdk or similar exists ('npm view "@ploy/sdk" version' returned E404). Add a minimal hosted API or SDK that lets a developer exercise core product actions (e.g., read a workspace, create a page) without local CLI install or OAuth browser flow, so lightweight evaluation is possible. Verify by having a fresh script call it successfully without local login. Event 73: ```text @ploy/sdk -> npm error code E404 ``` Event 48: ```text "title": "Ploy Migration Estimate API", "version": "1.0.0", "description": "Scan a public website and return a directional manual migration estimate plus the recommended Ploy plan and signup URL." ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/68759b76-dac5-47b4-a862-34088c3156fd) · [Read transcript](https://www.ax-check.com/ploy.ai/sessions/kimi.json) Quoted plan prices ($0, $50/mo, $300/mo, Custom) with each plan's credit allowance, taken from the fetched pricing page. Said credits are spent only during AI work and that a live site uses almost none. (seq 41, seq 130) #### End-to-end onboarding - **Onboarding needs a login**: Agent installed the real Ploy CLI (v0.17.0) and attempted device-code login against the hosted ploy.ai service, but authorization requires a human to approve the code in a browser or supply a developer token from the dashboard. No authenticated operation against the live product was ever completed; the agent instead wrote local example scripts that were never executed with real credentials. Event 85: ```text Open this URL on a device with a browser: https://ploy.ai/cli/auth/device?user_code=2UCB-EDU7 Confirm code 2UCB-EDU7. Waiting for authorization... exit: 0 ``` Event 87: ```text I'm blocked on authentication — completing this requires a Ploy account in a browser, which I can't create or approve myself. ``` Event 81: ```text Not logged in. Run `ploy login`. exit: 3 ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **CLI login requires human browser approval**: The Ploy CLI's device-code login flow printed a verification URL and code, but completing it requires a human to open a browser, log into an existing Ploy account, and approve the code. This is normal product authentication behavior, not a defect, but it stopped the agent from running any real commands against the hosted service. Event 85: ```text Open this URL on a device with a browser: https://ploy.ai/cli/auth/device?user_code=2UCB-EDU7 Confirm code 2UCB-EDU7. Waiting for authorization... ``` Event 87: ```text I'm blocked on authentication — completing this requires a Ploy account in a browser, which I can't create or approve myself. ``` - **Headless token also requires existing dashboard account**: The documented headless alternative (PLOY_API_TOKEN) requires creating a token inside workspace settings in the web dashboard, which itself requires a logged-in account — so there was no self-service path to obtain credentials without a human. Event 76: ```text Set PLOY_API_TOKEN to authenticate headlessly (CI, remote workers) instead of `ploy login`. Create one in workspace settings → Developer. ``` #### Suggested Changes - **Fix --help text for site init to note --dry-run is unsupported**: Running `ploy site init --name "hello-ploy-example" --dry-run` fails with a usage error even though the top-level CLI help advertises --dry-run as a general option for mutating commands. Update the `ploy site init --help` output (or top-level --dry-run description) to explicitly state that site init does not support it, and verify by re-running the command and confirming no exit-2 error. Event 98: ```text --dry-run is not supported for `site init`. It previews mutating commands only. exit: 2 ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/4a2db695-32fc-4aa9-8580-e12aa4fdc9f5) · [Read transcript](https://www.ax-check.com/ploy.ai/sessions/qwen.json) Confirmed no public pricing (pricing page 404, demo-only site). Called it quote-based and per-user, with the per-user part labelled a guess. No dollar figure given. (seq 34, seq 44) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No credentials were obtained or used. The agent searched for a Ploy API key in environment variables, found none, discovered the real product (ploy.io) is a sales-led identity governance SaaS with no public docs/API/pricing page and no self-serve signup, and correctly stopped without attempting any authenticated operation. Event 10: ```text env | grep -i -E "ploy|API|TOKEN|KEY" | sed 's/=.*/=/'; which ploy; npm ls -g --depth=0 2>/dev/null; pip list 2>/dev/null | grep -i ploy ``` Event 44: ```text No self-serve signup. The product is sales-led ("Get a demo," "Book a demo," white-glove onboarding); the app at `app.joinploy.com` requires a customer account. ``` Event 44: ```text No credentials in this environment (no Ploy API key/token anywhere), and its GitHub org (`join-ploy`) exposes no public repos I could use. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No public API, SDK, or docs for the real Ploy product**: Product behavior/business model issue, not agent error: the actual Ploy (ploy.io, identity governance SaaS) publishes no developer documentation or API reference. Requests to /docs, /api, and docs.ploy.io all returned 404, and the help center is end-user focused rather than developer focused. Event 34: ```text -- /docs: 404 Ploy ``` Event 34: ```text -- /api: 404 Ploy ``` Event 44: ```text No public SDK or API docs. `ploy.io/docs`, `/api`, and `docs.ploy.io` all 404. The only docs-ish resource is a help center (help.ploy.io) aimed at end users, not developers. ``` - **No self-serve signup or credentials available**: Missing credentials, rooted in product's sales-led go-to-market rather than a test-environment gap: the app requires a customer account via demo/sales process, and no API key existed anywhere in the sandbox environment. Event 12: ```text PI_GATEWAY_API_KEY= GIT_CONFIG_KEY_0= ``` Event 44: ```text No self-serve signup. The product is sales-led ("Get a demo," "Book a demo," white-glove onboarding); the app at `app.joinploy.com` requires a customer account. ``` - **Name collision with unrelated npm/PyPI packages**: Agent error avoided, not a true blocker: a search for 'ploy' on npm and PyPI returned unrelated packages (a git-push deploy router and a FreeBSD jail manager). The agent correctly identified these as irrelevant to the actual hosted product and did not waste effort building against them. Event 16: ```text {"_id":"ploy","_rev":"115-19155a5d8318fc1d108d7e66840785df","name":"ploy","description":"git push deploy http router, hosting branches on subdomains" ``` Event 44: ```text There is an npm package called `ploy` (an old self-hosted git-push deploy router) and a PyPI `ploy` (FreeBSD jail management), but neither is the hosted product you're asking about ``` #### Suggested Changes - **Publish a developer API reference at ploy.io/docs or docs.ploy.io**: Both ploy.io/docs and docs.ploy.io currently return 404. Adding a working API reference at one of these conventional URLs would let developers self-serve instead of hitting dead ends; verify by curling the URL and confirming a 200 with actual endpoint documentation. Event 34: ```text -- /docs: 404 Ploy ``` - **Add a pricing page at ploy.io/pricing**: Requesting ploy.io/pricing returns a 404, forcing prospective users into a sales/demo funnel with no visible pricing information. Publishing a pricing page (even tiered/contact-for-enterprise ranges) would let evaluators like this one answer basic pricing questions without a demo call; verify by checking /pricing returns 200 with pricing content. Event 34: ```text -- /pricing: 404 Ploy ``` - **Offer a self-serve API key or sandbox for developer evaluation**: The only path to product access found was sign-in at app.joinploy.com or booking a demo, with no API key issuance visible anywhere in the public site. Adding a self-serve trial or sandbox credential flow (e.g., a 'Get API key' button alongside 'Get a demo') would let developers test the product programmatically; verify by confirming a new account can obtain a working API key without sales contact. Event 30: ```text Log in Get a demo Meet Luna, the agent for identity governance ``` ### Task given to each agent Help me build a simple example using Ploy. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 100/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown (200) when requested with Accept: text/markdown. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt (200) indexes product, docs, MCP, CLI, skills, API and Ploybooks with descriptions. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt groups links under clear headings: Product, Tools, Comparisons, Company information. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt lists Ploy MCP, ploy-site skill, Ploy CLI, and Migration estimate API. ``` - **Pass** — A compact guide representation exists ```text Homepage returns text/markdown; docs quick-start and MCP pages also serve Markdown. ``` - **Pass** — A focused guide is directly retrievable ```text docs.ploy.ai/quick-start gives concrete setup steps, goals, and first-chat workflow. ``` - **Pass** — Equivalent instructions fit a token budget ```text Quick-start Markdown is 3320 tokens, well under the 8000-token budget. ``` - **Pass** — Product-docs links survive format changes ```text Homepage Markdown preserves docs, MCP, CLI, and Ploybooks links. ``` - **Pass** — The compact guide is independently actionable ```text Quick Start gives concrete setup steps, goal choices, first chat, review and publish flow. ``` - **Pass** — Install and next-step links resolve ```text Fetched install and next-step docs (quick-start, CLI install, MCP) resolve successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quick-start page gives concrete setup steps: website context, goals, first chat, publish. ``` - **Pass** — Installation commands are extractable ```text CLI install page shows curl install command and ploy login; CLI guide repeats it. ``` - **Pass** — Code examples are available without interaction ```text Install, migration and CLI pages show copyable shell, JSON and config code blocks inline. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text CLI docs state ploy login for interactive use and PLOY_API_TOKEN for headless CI auth. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders as static Markdown with all plan prices and features visible, no interaction needed. ``` - **Pass** — Prices are stated, not gated ```text Free $0, Starter $50/mo, Pro $300/mo stated openly; Enterprise custom. ``` - **Pass** — Pricing units and limits are explicit ```text Credits per plan, visitor enrichment limits, and custom domain families are explicit. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Gave Free/$50/$300/Custom tiers with credit allotments. Explained credits are spent only on active builds, plus the enrichment caps and domain-family rules. README wrongly puts some Starter features under Free. (seq 41, seq 86, README.md) Kimi K3: Quoted plan prices ($0, $50/mo, $300/mo, Custom) with each plan's credit allowance, taken from the fetched pricing page. Said credits are spent only during AI work and that a live site uses almost none. (seq 41, seq 130) Qwen 3.8 Max: Confirmed no public pricing (pricing page 404, demo-only site). Called it quote-based and per-user, with the per-user part labelled a guess. No dollar figure given. (seq 34, seq 44) This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text OpenAPI 3.1 spec for Ploy Migration Estimate API fetched at ploy.ai/agent-tools/openapi.json. ``` - **Pass** — An MCP server is documented and well-formed ```text Remote MCP at https://ploy.ai/api/mcp, Streamable HTTP, OAuth, per-client setup documented. ``` - **Pass** — A CLI install path is documented ```text CLI install documented: curl install.sh, ploy login, standalone binary, macOS/Linux. ``` - **Skipped** — SDK packages resolve on their registries ```text No registry lookup for a Ploy SDK or packaged CLI package was supplied. ``` - **Pass** — Agent skills are published ```text ploy-site agent skill published in Ploy-AI/ploy-tools with install methods and SKILL.md. ``` [Full report data](https://www.ax-check.com/ploy.ai/report.json)