# AX Check: agentmail.to Checked 2026-10-07. AgentMail's docs, quickstart and pricing are all agent-ready. All 23 checklist items passed: clear onboarding docs, install commands, code samples, API/MCP/CLI/SDK access, and openly stated pricing tiers. ## Onboarded with real credentials ## Coding sessions All three independent sessions (DeepSeek V4.1 Flash, Kimi K3, Qwen 3.8 Max) completed the task and produced full pricing breakdowns pulled directly from the pricing page, each listing Free, Developer, Startup and Enterprise tiers with concrete limits like inbox counts, emails per month, and storage, with no login wall blocking access to this information. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/57947dfa-5dc1-4e76-92a8-96222cf90c50) · [Read transcript](https://www.ax-check.com/agentmail.to/sessions/deepseek.json) Final output gives a full pricing table (Free $0, Developer $20/mo, Startup $200/mo, Enterprise custom) scraped live from agentmail.to/pricing (seq 32-46), with named assumptions like plan limits, inbox counts, and add-on costs. #### End-to-end onboarding - **Onboarding needs a login**: The agent researched AgentMail's docs, pricing, and SDK thoroughly and built a working example project, but never obtained a real API key. It only tested with a known-invalid placeholder key (am_placeholder) and unauthenticated requests, both of which returned 403/401 errors from the live API. It explicitly declined to run the self-serve agent sign-up flow (POST /agent/sign-up) that could have produced a real key without human intervention, citing side effects, and instead asked the user to supply a key or an email address. Event 75: ```text AgentMailError status: 403 body: {"message":"Forbidden"} ``` Event 100: ```text === run with placeholder key (live API call) === AgentMail API error 403: Forbidden ``` Event 106: ```text I have **no AgentMail API key**, so I could not create a real inbox or send a live email — only verify up to the auth boundary. To finish, paste a key from https://console.agentmail.to into `.env`. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No API key obtained, so no authenticated inbox/send call completed**: The agent never ran AgentMail's documented self-serve agent sign-up (POST /agent/sign-up), which the quickstart says returns an API key without needing a Console login. Instead it tested only with no key and a placeholder key, both rejected by the live API (401/403). This is a self-imposed scope limit (agent chose not to execute the sign-up flow) rather than a product defect, since the docs show a credential path requiring only a human email to be provided. Event 100: ```text === run with placeholder key (live API call) === AgentMail API error 403: Forbidden ``` Event 106: ```text I didn't create an account on your behalf since that has side effects and needs an email you control — tell me if you'd like me to walk that flow with an address you provide. ``` #### Suggested Changes None identified in this transcript. ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/56490b8f-20a5-42fd-86ec-65eacbcc759e) · [Read transcript](https://www.ax-check.com/agentmail.to/sessions/kimi.json) Final output lists Free/Developer/Startup/Enterprise tiers with concrete limits (inboxes, emails/day, storage) sourced from agentmail.to/pricing, tying the $0 figure to the free-tier/unverified-org assumptions it was actually operating under. #### End-to-end onboarding - **Onboarded with real credentials**: The agent obtained live AgentMail credentials entirely on its own via the documented self-service agent sign-up endpoint (no human email, no human-supplied key), then used that API key through the official SDK to perform a real authenticated operation against the hosted product: listing the organization's actual inbox and attempting to send a message (rejected only by the product's anti-abuse allow-list policy, not an auth failure). This satisfies both credential acquisition and authenticated operation against the real hosted API. Event 45: ```text "organization_id":"be3879f7-8690-43c1-886b-6e576953404b","inbox_id":"pi-demo-agent326@agentmail.to","email":"pi-demo-agent326@agentmail.to","api_key":"am_us_b86605c59eabcdef5010f496ea77e37e1205dde76c47ec7f37b61da2911e045a" ``` Event 65: ```text Using existing inbox: pi-demo-agent326@agentmail.to ``` Event 65: ```text Total inboxes: ['pi-demo-agent326@agentmail.to'] ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Sending email blocked pending human verification**: Product behavior, not agent error: self-signup agent accounts are receive-only until a human email is attached and verified via OTP. The send call correctly failed with a structured error explaining the allow-list restriction tied to unverified agent orgs. This is an intentional anti-abuse control requiring human involvement, so it is a session limitation rather than a product defect. Event 65: ```text message='Message rejected: Recipient(s) blocked: recipient@example.com (not in allow list)' fix="Remove these recipients from the request, or add each missing recipient to the send allow list with POST /v0/lists/send/allow ... unless this organization is an agent org that has not completed verification, in which case sending is restricted to the human's email and list entries cannot be created yet; complete POST /v0/agent/verify instead" ``` - **Inbox creation limit hit pre-verification**: Product behavior: creating a new inbox failed with a 403 limit_exceeded error because unverified agent orgs are capped at 1 inbox instead of the free tier's 3. The agent recovered immediately by reusing the already-created inbox instead of creating a new one, so this did not block overall progress. Event 58: ```text body: {'name': 'LimitExceededError', 'code': 'limit_exceeded', 'message': 'Inbox limit exceeded', 'fix': 'This organization is not verified yet, so its limits are the pre-verification defaults. Ask the human for the verification code we emailed them and call POST /v0/agent/verify with it.' ``` #### Suggested Changes - **Document the pricing.md 404 or add a redirect**: Fetching https://docs.agentmail.to/pricing.md returned a 'Page Not Found' with unrelated suggested links (Permissions, Introduction, Reply & Attachment Extraction), forcing a fallback to scraping the marketing site HTML for pricing details. Add a pricing.md doc page or redirect it to the actual pricing page so agents following the documented '.md' convention from llms.txt can get pricing info directly; verify by re-fetching docs.agentmail.to/pricing.md and confirming it returns pricing content instead of a 404. Event 30: ```text # Page Not Found This page does not exist. ## Similar pages - [Permissions](https://docs.agentmail.to/permissions.md) - [Introduction](https://docs.agentmail.to/introduction.md) - [Reply & Attachment Extraction](https://docs.agentmail.to/reply-extraction.md) ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/9607dd63-eb77-444b-96f1-f6e9049238a8) · [Read transcript](https://www.ax-check.com/agentmail.to/sessions/qwen.json) Final output lists Free/Developer/Startup/Enterprise tiers with concrete limits (3 vs 10 vs 150 inboxes, emails/mo, overage $2 add-ons, 20% yearly discount) pulled from agentmail.to/pricing, tying cost to plan/workload assumptions rather than a bare number. #### End-to-end onboarding - **Onboarded with real credentials**: Agent self-registered via AgentMail's programmatic sign-up endpoint with no human help, receiving a real API key and inbox id. It then used that key via the Python SDK against the live hosted API: listing inboxes (returned the real inbox pi-demo-agent@agentmail.to) and attempting a send/list, both producing authenticated, non-mock server responses including real product-side rate-limit enforcement. This satisfies credential acquisition plus an authenticated operation, even though sending stayed gated by verification. Event 56: ```text curl -s -X POST https://api.agentmail.to/agent/sign-up -H "Content-Type: application/json" -d '{"username": "pi-demo-agent"}' | python3 -m json.tool ``` Event 58: ```text "inbox_id": "pi-demo-agent@agentmail.to", "email": "pi-demo-agent@agentmail.to", "api_key": "am_us_4deb9ab ``` Event 118: ```text Inboxes: ['pi-demo-agent@agentmail.to'] Send blocked (expected until verification): MessageRejectedError - Message rejected: Recipient(s) blocked: your-email@example.com (not in allow list) ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Sending email blocked pending human verification**: Product behavior, not agent error: AgentMail requires a human to verify the agent-created organization (emailed OTP or console claim) before it can send email or create more than one inbox. The agent diagnosed this from the API's own error messages and worked around it by demonstrating list/receive operations instead. Event 86: ```text send blocked: MessageRejectedError name='MessageRejectedError' code='message_rejected' message='Message rejected: Recipient(s) blocked: pi-demo-agent@agentmail.to (not in allow list)' ``` Event 90: ```text This organization has not completed agent verification, which gates 'list_entry_create' at every scope ``` - **Inbox creation capped at 1 for unverified org**: Test environment/product limit: the free unverified sign-up tier caps inbox creation at 1, so a second create() call failed with a 403 limit_exceeded error. Minor agent error contributed since it first tried creating a second inbox instead of reusing the existing one, but it recovered quickly. Event 69: ```text agentmail.core.api_error.ApiError: headers: {...} status_code: 403, body: {'name': 'LimitExceededError', 'code': 'limit_exceeded', 'message': 'Inbox limit exceeded' ``` - **SDK error-handling friction (agent error, self-corrected)**: Agent assumed a dict-like error body and an importable ApiError from the top-level package, both wrong for this SDK version, causing two failed script runs before fixing the import path and attribute access. Event 98: ```text ImportError: cannot import name 'ApiError' from 'agentmail' (/opt/freestyle/python/lib/python3.12/site-packages/agentmail/__init__.py) ``` Event 110: ```text AttributeError: 'ErrorResponse' object has no attribute 'get' ``` #### Suggested Changes - **Clarify SDK error body access pattern in quickstart docs**: On the Quickstart page's Python code sample, add a short snippet showing how to read fields off the SDK's typed ApiError body (e.g. e.body.name, e.body.message) since the body is a pydantic model, not a dict. Verify by running the sample as-is and confirming no AttributeError on a rejected send. Event 110: ```text AttributeError: 'ErrorResponse' object has no attribute 'get' ``` - **Surface the 1-inbox pre-verification cap in sign-up response instructions**: In the POST /agent/sign-up response 'instructions' text, explicitly state the 1-inbox limit before verification, not just that sending is blocked. This lets agents avoid an extra create() call that fails with limit_exceeded. Verify by checking a fresh sign-up response mentions the inbox cap alongside the send restriction. Event 58: ```text It can receive email now. It cannot send email yet, because no human is attached to your account. ``` ### Task given to each agent Help me build a simple example using AgentMail. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 100/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown with 200 when Accept: text/markdown was sent. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs, API reference, MCP, SDKs, quickstarts and setup guides. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt gives ordered agent onboarding steps and when-to-use guidance. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt covers REST API, hosted MCP server, and Claude skills. ``` - **Pass** — A compact guide representation exists ```text Homepage returns text/markdown; docs pages serve .md versions like quickstart.md. ``` - **Pass** — A focused guide is directly retrievable ```text llms.txt and quickstart.md give directly retrievable, focused agent onboarding steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text llms.txt is 6539 tokens, under 8000, and preserves the essential workflow. ``` - **Pass** — Product-docs links survive format changes ```text Homepage Markdown keeps Documentation, API Reference, Console, Pricing and Enterprise links. ``` - **Pass** — The compact guide is independently actionable ```text SKILL.md gives install, client init, inbox create, send, webhooks, gotchas — independently actionable. ``` - **Pass** — Install and next-step links resolve ```text Quickstart install (pip/npm/CLI) and next-step links (WebSockets, webhooks, API ref) fetched successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quickstart page gives concrete first steps: sign up, verify OTP, create inbox, send email. ``` - **Pass** — Installation commands are extractable ```text Install commands extractable: pip install agentmail, npm install agentmail, npm install -g agentmail-cli. ``` - **Pass** — Code examples are available without interaction ```text Full Python, TypeScript and cURL code examples shown inline without interaction. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Auth explicit: AGENTMAIL_API_KEY env var, Console key generation, agent sign-up OTP flow. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders as plain Markdown with all plan prices and limits, no interaction needed. ``` - **Pass** — Prices are stated, not gated ```text Free $0, Developer $20/mo, Startup $200/mo stated openly; Enterprise custom. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: inboxes, emails/month, emails/day, storage GB, custom domains per tier. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Final output gives a full pricing table (Free $0, Developer $20/mo, Startup $200/mo, Enterprise custom) scraped live from agentmail.to/pricing (seq 32-46), with named assumptions like plan limits, inbox counts, and add-on costs. Kimi K3: Final output lists Free/Developer/Startup/Enterprise tiers with concrete limits (inboxes, emails/day, storage) sourced from agentmail.to/pricing, tying the $0 figure to the free-tier/unverified-org assumptions it was actually operating under. Qwen 3.8 Max: Final output lists Free/Developer/Startup/Enterprise tiers with concrete limits (3 vs 10 vs 150 inboxes, emails/mo, overage $2 add-ons, 20% yearly discount) pulled from agentmail.to/pricing, tying cost to plan/workload assumptions rather than a bare number. This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text API reference page fetched at docs.agentmail.to/api-reference with base URL and SDK links. ``` - **Pass** — An MCP server is documented and well-formed ```text Hosted MCP server documented with endpoint, OAuth/API-key auth, and 37 named tools. ``` - **Pass** — A CLI install path is documented ```text CLI install documented: npm install -g agentmail-cli, with agentmail commands. ``` - **Pass** — SDK packages resolve on their registries ```text PyPI agentmail and npm agentmail registry lookups both returned HTTP 200. ``` - **Pass** — Agent skills are published ```text AgentMail skill published as SKILL.md with workflows and reference files. ``` [Full report data](https://www.ax-check.com/agentmail.to/report.json)