# AX Check: relayfi.com Checked 2026-09-19. Pricing is posted clearly, but agents can't find it Relay states exact plan prices and limits on its pricing page, yet all 3 independent agent sessions failed to report the actual figures. ## Onboarding needs a login ## Coding sessions Three independent sessions tried to determine Relay's pricing. One said it could not quote current tier prices, another reported pricing as free based on incorrect assumptions, and a third said the pricing page returned an error and declined to guess numbers. ### DeepSeek V4 Pro [View public run](https://agents.withgauge.com/p/runs/d8f5bd4b-dd29-48f4-8614-cb8f9755313b) · [Read transcript](https://www.ax-check.com/relayfi.com/sessions/deepseek.json) Final output explicitly says 'I can't quote current per-tier prices' (seq 52) — it lists billing units (steps, AI credits, free/paid tiers) but never states an actual cost figure, so the rubric's core requirement (stating what the product would cost) is unmet. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: The agent never attempted to obtain credentials or make an authenticated call. It discovered relay.app returns a shutdown notice (signups disabled, docs/API 404), correctly concluded no live developer workflow exists, and stopped without pursuing a login, API key, or SDK call of any kind. Event 30: ```text Starting July 16 New signups and free-to-paid upgrades are now turned off. ``` Event 52: ```text I cannot get an API key or call a hosted API (all accounts and associated data are deleted, and the API/docs are gone). ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Relay.app product is fully shut down**: This is product/business behavior, not agent error or a test-environment issue. The live site itself confirms signups and upgrades were disabled and both free and paid accounts are being (or have been) deleted, so no account, API key, or hosted endpoint can be created. The agent verified this directly from the vendor's own shutdown notice before stopping. Event 30: ```text We have difficult news to share: Relay.app will be shutting down on August 15, 2026 for free users and September 14, 2026 for paying customers. ``` Event 30: ```text After Mon Sep 14 at 23:59 PT , paid accounts and all associated data will be permanently deleted. ``` - **Docs, pricing, and API pages all return 404 or redirect to the shutdown banner**: Product behavior (post-shutdown site pruning), not agent error: the agent correctly probed the real documented paths (/docs, /pricing, /developers, /api) and each one now serves the shutdown notice or a 404, confirming no live developer surface remains. Event 36: ```text === https://relay.app/pricing === HTTP 404 final=https://relay.app/pricing Relay.app is shutting down Relay.app is shutting down. Read the announcement . ``` Event 36: ```text === https://relay.app/api === HTTP 404 final=https://relay.app/api Relay.app is shutting down Relay.app is shutting down. Read the announcement . ``` #### Suggested Changes None identified in this transcript. ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/334a2580-c46f-4861-8255-d79c8d0a243d) · [Read transcript](https://www.ax-check.com/relayfi.com/sessions/kimi.json) Final output states pricing is 'Free' and grounds it in stated assumptions — MIT open-source license, no hosted Relay service/usage metering, costs instead depend on the chosen GraphQL backend and own hosting/CI. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No product credentials were obtained or used. The agent interpreted 'Relay' as Meta's open-source GraphQL client library (no account, no API key, no hosted service) rather than the SaaS product 'Relay.app', which it flagged as requiring credentials it could not create. It built against a public third-party GraphQL demo API (GraphQLZero) instead of any Relay-branded hosted product, so there is no authenticated product operation to verify. Event 95: ```text ran the compiled query live from Node: got real data back (`User: Leanne Graham ` plus her posts) ``` Event 95: ```text If you actually meant **Relay.app** (the workflow-automation SaaS): say so plainly — that one requires an account/API key I can't create from here, so I'd be blocked on credentials before building anything. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Ambiguous product name resolved without confirming with user**: Two very different products are named Relay: Meta's open-source GraphQL client, and Relay.app, a workflow-automation SaaS. The agent silently chose the open-source library, built against a public demo GraphQL API rather than any Relay-branded hosted service, and only flagged the SaaS possibility as a caveat at the end. This is an agent judgment call, not a product defect, but the pricing/manageability assessment given may not answer what the user intended. Event 95: ```text using **Relay** (Meta's GraphQL client — I assumed this one, not Relay.app) ``` Event 95: ```text that one requires an account/API key I can't create from here, so I'd be blocked on credentials before building anything. ``` - **npm install timed out on first attempt**: Installing dependencies exceeded the 180-second timeout on the first try; a test-environment/tooling slowness issue, not a product defect. Agent recovered by re-running just the missing dev dependencies with a longer timeout, which succeeded. Event 27: ```text Command timed out after 180 seconds ``` Event 39: ```text relay-compiler vite ``` #### Suggested Changes - **Disambiguate 'Relay' branding for developers arriving via search or prompt**: A prompt simply asking about 'Relay' pricing led the agent to default to the open-source GraphQL client rather than the Relay.app SaaS product, surfacing the ambiguity only as an afterthought. If Relay.app is the intended product, add disambiguation copy so generic 'Relay' queries do not resolve to the unrelated library. Check by testing whether a fresh session given only the word 'Relay' lands on the SaaS product's docs rather than the GraphQL client's site. Event 95: ```text I assumed this one, not Relay.app ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/56f56a07-33c4-4ffa-9690-1f1c1f003868) · [Read transcript](https://www.ax-check.com/relayfi.com/sessions/qwen.json) Agent explicitly declines to give a cost figure — 'the pricing page itself 404s, so I can't confirm actual tier prices or limits, and I won't guess numbers' (seq 40) — describing only the billing mechanism (Paddle, steps/credits) with no stated price. #### End-to-end onboarding - **Onboarding needs a login**: No self-service onboarding occurred. The agent never obtained or used any Relay.app credentials; the only credential present in the sandbox was an unrelated PI_GATEWAY_API_KEY. The agent discovered Relay.app has disabled new signups entirely as part of a company shutdown, so there was no path to obtain real product credentials without human intervention, and no authenticated operation was attempted or possible. Event 6: ```text PI_GATEWAY_API_KEY= ``` Event 30: ```text Starting July 16 New signups and free-to-paid upgrades are now turned off. ``` Event 40: ```text Signups are closed, so no account can be created — and there are no Relay credentials in this environment (only `PI_GATEWAY_API_KEY`). ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Relay.app has shut down and blocks all new signups**: This is a product/business-state blocker, not an agent error or missing credential issue in the usual sense. The live site at relay.app, its docs subdomain, and its app subdomain all redirect to a single shutdown announcement page. New signups and free-to-paid upgrades were disabled starting July 16, and all accounts and data are slated for permanent deletion (free by Aug 15, paid by Sep 14). This made it impossible to create a developer account, find pricing details, or reference API documentation. Event 30: ```text Relay.app is shutting down Log in An important update on Relay.app We have difficult news to share: Relay.app will be shutting down on August 15, 2026 for free users and September 14, 2026 for paying customers. ``` Event 38: ```text https://docs.relay.app/api-reference -> 200 https://relay.app/ Relay.app is shutting down ``` Event 37: ```text <title>Relay.app is shutting down ``` - **API hostname does not resolve**: Test-environment/product infrastructure issue: api.relay.app has no DNS entry, so even if credentials existed, the API could not be reached from the sandbox. This compounds the shutdown blocker and confirms there is no live API surface to build against. Event 37: ```text NO DNS for run/api.relay.app ``` Event 15: ```text curl: (6) Could not resolve host: api.relay.app 000 ``` #### Suggested Changes - **Return explicit shutdown status codes on docs and API subdomains**: docs.relay.app currently 200-redirects silently to the marketing shutdown page, and api.relay.app fails to resolve at all, giving automated tooling no machine-readable signal of discontinuation short of full-page text scraping. Update these subdomains to return a distinct status (e.g., 410 Gone) referencing the shutdown. Verify by curling https://docs.relay.app/api-reference and confirming a non-200 response instead of today's 200 masquerading as normal content. Event 38: ```text https://docs.relay.app/api-reference -> 200 https://relay.app/ Relay.app is shutting down ``` ### Task given to each agent Help me build a simple example using Relay. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: B · 71/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html even when text/markdown was requested; no Markdown representation offered. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt lists primary pages, key guides, solutions and optional resources with descriptive links. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt organizes links under Primary Pages, Key Guides, Solutions and Optional headings. ``` - **Skipped** — llms.txt mentions offered API, MCP, and skills ```text llms.txt mentions integrations but no API, MCP or skills surfaces are offered or referenced. ``` - **Pass** — A compact guide representation exists ```text llms.txt is a compact Markdown guide to Relay's pages, plans and features. ``` - **Pass** — A focused guide is directly retrievable ```text llms.txt fetched directly at /llms.txt as text/plain Markdown, 506 tokens. ``` - **Pass** — Equivalent instructions fit a token budget ```text llms.txt is 506 tokens, well under the 8000-token budget. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be measured. ``` - **Skipped** — The compact guide is independently actionable ```text No compact agent guide fetched; only llms.txt index and a webinar page, not an actionable quickstart. ``` - **Skipped** — Install and next-step links resolve ```text No install or next-step links were sampled; homepage links were not fetched to verify resolution. ``` ### Onboarding - **Skipped** — Docs lead to a relevant quickstart ```text Relay is a business banking product; no developer quickstart or docs were fetched. ``` - **Skipped** — Installation commands are extractable ```text No developer installation or CLI setup instructions appear in the fetched pages. ``` - **Skipped** — Code examples are available without interaction ```text Fetched pages are marketing and webinar content, with no developer code examples. ``` - **Skipped** — Prerequisites and auth boundaries are explicit ```text No developer API or auth documentation was fetched for this banking product. ``` ### Pricing - **Failed** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 2 fell short. DeepSeek V4 Pro: Final output explicitly says 'I can't quote current per-tier prices' (seq 52) — it lists billing units (steps, AI credits, free/paid tiers) but never states an actual cost figure, so the rubric's core requirement (stating what the product would cost) is unmet. Kimi K3: Final output states pricing is 'Free' and grounds it in stated assumptions — MIT open-source license, no hosted Relay service/usage metering, costs instead depend on the chosen GraphQL backend and own hosting/CI. Qwen 3.8 Max: Agent explicitly declines to give a cost figure — 'the pricing page itself 404s, so I can't confirm actual tier prices or limits, and I won't guess numbers' (seq 40) — describing only the billing mechanism (Paddle, steps/credits) with no stated price. This behavioural item does not affect the fast grade. ``` - **Pass** — Pricing is readable without interaction ```text Pricing page renders plan tiers and prices as static text without interaction. ``` - **Pass** — Prices are stated, not gated ```text Starter $0, Grow $30, Scale $120/$90 per month stated openly. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: per month, 20/50 checking accounts, 50 cards, APY and cash-back rates. ``` ### Activation - **Skipped** — An API reference or OpenAPI spec is reachable ```text No API reference or OpenAPI spec appears in the fetched Relay pages. ``` - **Skipped** — An MCP server is documented and well-formed ```text No MCP server is documented anywhere in the fetched Relay content. ``` - **Skipped** — A CLI install path is documented ```text No CLI install path is documented in the fetched Relay pages. ``` - **Skipped** — SDK packages resolve on their registries ```text No SDK or developer package registry result was supplied for Relay. ``` - **Skipped** — Agent skills are published ```text No agent skills are published in the fetched Relay content. ``` [Full report data](https://www.ax-check.com/relayfi.com/report.json)