# AX Check: comail.io Checked 2026-09-19. Comail's docs give agents a clear MCP setup and upfront pricing llms.txt spells out MCP config, auth, and $39/month-per-inbox pricing (up to 10 agents) with no gating; homepage just won't serve Markdown directly. ## Onboarding needs a login ## Coding sessions Of three independent sessions, one (DeepSeek V4 Pro) did not complete. The other two (Kimi K3, Qwen 3.8 Max) both finished and correctly reported pricing as $39/month per inbox for up to 10 agents, noting it as an early-bird private-beta rate with billing starting at inbox provisioning. ### DeepSeek V4 Pro [View public run](https://agents.withgauge.com/p/runs/cdc94248-dec0-4263-99a1-129129515332) · [Read transcript](https://www.ax-check.com/comail.io/sessions/deepseek.json) #### End-to-end onboarding - **Onboarding needs a login**: The agent never obtained real comail credentials. It researched the product entirely via public web pages (docs/llms.txt, homepage), then wrote a local MCP client script but explicitly could not run it because it lacks an API key or MCP URL. It stated plainly that private-beta admission, mailbox owner OAuth, and an API key issued by the owner in the dashboard are required and it cannot self-serve those. No authenticated call against the hosted comail MCP server occurred. Event 213: ```text There **is** an actionable developer workflow, but I could **not** run a live end-to-end example in this session because it requires: 1. **Private-beta admission** (waitlist). 2. The mailbox **owner doing Google OAuth in their browser** and creating an agent to get the URL + key. 3. A real **API key**, which only the owner can issue in the dashboard. ``` Event 202: ```text Set COMPAIL_MCP_URL and COMPAIL_API_KEY. Example key format: cml_live_... exit=1 ``` Event 213: ```text I can't self-serve those credentials, so I prepared the example but couldn't execute it against comail. ``` #### Hallucinated URLs - **Guessed docs subdomain and paths returned 404/no DNS**: The agent tried numerous guessed documentation URLs (docs.comail.io and many comail.io/docs* and API-reference-style paths) that all failed to resolve or 404'd. These look like guesses based on common doc conventions rather than links the agent found referenced anywhere, and the guessing failed to surface any real docs endpoint. Event 124: ```text == https://comail.io/docs == 404 https://comail.io/docs == https://docs.comail.io == 000 https://docs.comail.io/ == https://comail.io/docs/send == 404 https://comail.io/docs/send == https://comail.io/docs/api-keys == 404 https://comail.io/docs/api-keys ``` Event 163: ```text 404: /api-keys 404: /http-send-api 404: /send-api 404: /reference 404: /docs/api-keys 404: /docs/http-send-api 404: /docs/send 404: /api 404: /send-message 404: /create-draft 404: /search-threads 404: /get-thread 404: /notifications 404: /api-reference 404: /http-api 404: /rest-api 404: /endpoints ``` #### Blockers - **Private beta gating blocks any live credential or API call**: comail is described as private beta requiring waitlist admission, then owner-driven Google OAuth and dashboard-issued API keys. This is normal product access control (not a bug), but it fully prevented the agent from running its example against the real hosted service within this session. This is a session/product-access limitation rather than a defect. Event 213: ```text 1. **Private-beta admission** (waitlist). 2. The mailbox **owner doing Google OAuth in their browser** and creating an agent to get the URL + key. 3. A real **API key**, which only the owner can issue in the dashboard. ``` - **No public developer docs subdomain reachable**: Searches referenced a 'comail docs' HTTP Send API page and API keys page, but docs.comail.io does not resolve and no /docs paths exist on the main site; the agent had to rely solely on the llms.txt file at the root for API/tool details. This is either a documented-but-broken link or docs hosted elsewhere not discovered; ambiguous, so not claimed as hallucination but noted as friction. Event 27: ```text API keys | comail docs Does Gmail API cost money? - Mike Gravel Does Gmail HTTP Send API | comail docs HTTP POST /v1/send: JSON request and response, error codes, retry guidance. ``` Event 158: ```text == docs.comail.io == (no DNS) ``` #### Suggested Changes - **Publish a resolvable docs subdomain or in-site docs path**: Search engines surfaced snippets referencing 'API keys | comail docs' and 'HTTP Send API | comail docs', but docs.comail.io does not resolve via DNS and comail.io/docs 404s. Publish the docs at a real, linked URL (e.g. docs.comail.io or comail.io/docs) and confirm by curling it directly and checking that primary site navigation links to it. Event 27: ```text HTTP Send API | comail docs HTTP POST /v1/send: JSON request and response, error codes, retry guidance. ``` Event 158: ```text == docs.comail.io == (no DNS) ``` - **Add a self-serve or sandbox credential path for evaluation**: Because comail requires waitlist admission plus owner-driven Google OAuth and dashboard-issued keys, a first-time developer cannot obtain any credential or test the MCP tools without a human already owning a connected mailbox. Consider a sandbox/demo agent with a stub mailbox so a new developer can exercise list_mailboxes/search_threads end-to-end before committing to OAuth. Verify by having a fresh, unauthenticated session attempt to acquire and use a key without human intervention. Event 213: ```text I can't self-serve those credentials, so I prepared the example but couldn't execute it against comail. ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/0421e5a2-81f8-4eda-b57d-b2609246aa22) · [Read transcript](https://www.ax-check.com/comail.io/sessions/kimi.json) Agent quotes $39/month per inbox covering up to 10 agents and calls out it's the early-bird private-beta rate with billing starting at inbox provisioning (seq 35), i.e. names the plan/tier assumptions behind the number. #### End-to-end onboarding - **Onboarding needs a login**: The agent could not obtain comail credentials on its own. comail is in private beta and requires a human to apply via a waitlist, get accepted, authorize Google OAuth from the web dashboard, and manually create an agent to get a bearer key. The agent wrote an example script but never executed it against the live product because it had no key. No authenticated tool call (list_mailboxes, search_threads, etc.) was ever made against the hosted comail API. Event 35: ```text provisioning requires a human to (1) be accepted, (2) authorize a Google account via OAuth from the web dashboard, and (3) create an agent and get its endpoint URL + `cml_live_...` key. None of that is self-serve or scriptable, so I have no credentials to exercise. ``` Event 43: ```text The one caveat: day-zero setup (beta acceptance, Google OAuth, agent creation, applying the first `co:` label) is owner-only work in the dashboard... which is rather the point of the product. ``` Event 37: ```text Set COMAIL_URL=... COMAIL_KEY=cml_live_... python comail_example.py ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Private beta gate blocks all live testing**: comail requires manual acceptance into a private beta, Google OAuth authorization from the owner's web dashboard, and manual agent/key creation before any API or MCP access is possible. This is a product design choice (gated beta with owner-only provisioning), not an agent error or environment issue. The agent correctly identified it had no path to self-serve credentials and stopped rather than fabricating a workaround. Event 35: ```text comail is in **private beta, onboarding in batches** — access requires applying at `comail.io/#waitlist`, and provisioning requires a human to (1) be accepted, (2) authorize a Google account via OAuth from the web dashboard, and (3) create an agent and get its endpoint URL + `cml_live_...` key. ``` #### Suggested Changes - **Offer a sandbox or demo agent key for evaluation**: Because comail requires beta acceptance, OAuth, and manual agent creation before any code can be exercised, developers evaluating the product (including automated agents) cannot verify the SDK/MCP workflow at all. Consider a scoped demo credential or sandbox mailbox reachable without the full beta application process, and confirm it works by having a fresh, uncredentialed session successfully call list_mailboxes or search_threads without waiting on manual dashboard steps. Event 35: ```text None of that is self-serve or scriptable, so I have no credentials to exercise. ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/bbe65e37-745f-49ae-971b-af984d739844) · [Read transcript](https://www.ax-check.com/comail.io/sessions/qwen.json) Final output states $39/month per inbox (up to 10 agents) and explicitly names it as the 'early-bird private-beta rate' with billing starting at inbox provisioning — figure paired with assumptions. #### End-to-end onboarding - **Onboarding needs a login**: Agent could not obtain any comail credentials on its own. It found no COMAIL_* env vars, no SDK/package on npm or PyPI, and confirmed via the product's own llms.txt that keys are issued only from a human-operated web dashboard after a manual Google OAuth grant, gated behind a private-beta waitlist. No authenticated API call was ever made; the agent stopped after read-only discovery. Event 8: ```text env | grep -i -E "comail|PI_" | head -30 ``` Event 28: ```text Agents do not. Each agent is issued its own MCP endpoint URL and its own bearer key, and authenticates to comail rather than to Google. ``` Event 34: ```text Keys cannot be self-served. They're issued per-agent from the web dashboard, after the mailbox owner completes an OAuth grant on their Google account from that dashboard. No signup flow, no CLI, no `comail init`. ``` Event 34: ```text Private beta, waitlist-gated. Access requires applying at `https://comail.io/#waitlist` and waiting to be onboarded in batches. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-serve credentials — dashboard + OAuth + waitlist required**: Product behavior, not agent error. comail's own documentation (llms.txt) confirms keys are only issued from a human-operated web dashboard after the mailbox owner does a manual Google OAuth grant, and the product is in private beta requiring a waitlist application. This makes autonomous, credential-based onboarding impossible in this session. Event 28: ```text The mailbox owner grants comail an OAuth authorization on their Google account, once, from the web dashboard. ``` Event 34: ```text Private beta, onboarding in batches. Applications are taken at ``` - **No SDK or public REST API exists**: Product design limitation. The agent confirmed via npm and PyPI searches that no comail package exists, and the product's own docs state explicitly that the only interface is MCP over HTTP with no SDK. This narrowed the 'use the hosted product through its SDK or API' request to a dead end. Event 12: ```text npm error 404 npm error 404 Not Found - GET https://registry.npmjs.org/comail - Not found ``` Event 28: ```text Standard Model Context Protocol over HTTP. Any MCP client can connect: Claude, Cursor, or a custom agent. There is no comail SDK. ``` - **Docs subdomain and API subdomain unreachable**: Test environment or product configuration issue (unclear which). Requests to docs.comail.io, api.comail.io, and mcp.comail.io all failed to resolve (curl exit code 6, DNS failure), while comail.io itself and comail.io/llms.txt worked fine. This is not proven to be a hallucinated URL since these are plausible standard subdomain conventions, but it did block further exploration of any hosted API surface. Event 32: ```text https://docs.comail.io -> 000 https://api.comail.io -> 000 https://mcp.comail.io/ -> 000 Command exited with code 6 ``` #### Suggested Changes - **Add a public docs page reachable at comail.io/docs**: Requesting https://comail.io/docs returned a 404 while the actual product specification lives only at /llms.txt, an unlinked machine-readable file. Publish a human-facing docs page linked from the main site nav (the crawled page only exposed nav anchors like #pricing, #features, #how) so prospective integrators are not forced to discover /llms.txt by guessing. Verify by confirming GET https://comail.io/docs returns 200 and links to the MCP interface section. Event 24: ```text href="#agents" href="#explicit" href="#features" href="#how" href="#main" href="#notifications" href="#pricing" href="#waitlist" ``` Event 32: ```text https://comail.io/docs -> 404 ``` - **Offer a sandbox or demo mailbox for waitlisted developers**: Because credential issuance requires a human dashboard OAuth grant and a waitlist, developers cannot evaluate the MCP tools (search_threads, create_draft, etc.) hands-on before approval. Consider a demo agent/mailbox with pre-seeded threads that new signups can query immediately, and confirm success by having a fresh signup call list_mailboxes and search_threads without owner intervention. Event 34: ```text No signup flow, no CLI, no `comail init`. ``` Event 28: ```text Private beta. ``` ### Task given to each agent Help me build a simple example using comail. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: B · 83/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html even when Markdown was requested via Accept header. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt is a detailed, well-organized index covering architecture, access model, MCP tools, and pricing. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt provides clear navigation via sectioned headings and a waitlist link. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt documents the MCP interface, management MCP, and states no comail SDK exists. ``` - **Pass** — A compact guide representation exists ```text llms.txt is a standalone Markdown guide covering architecture, access model, MCP tools and pricing. ``` - **Pass** — A focused guide is directly retrievable ```text llms.txt fetched directly at comail.io/llms.txt with 200 and full Markdown body. ``` - **Pass** — Equivalent instructions fit a token budget ```text llms.txt is 3129 tokens, well under the 8000-token budget, and self-contained. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be measured. ``` - **Pass** — The compact guide is independently actionable ```text llms.txt gives a complete, self-contained workflow: MCP config, tools, permissions, notifications, pricing. ``` - **Skipped** — Install and next-step links resolve ```text Homepage links only llms.txt and privacy; no install or next-step links were sampled. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text llms.txt gives concrete MCP setup: config JSON, endpoint URL, bearer key, and tool list. ``` - **Pass** — Installation commands are extractable ```text llms.txt shows extractable MCP client config JSON and curl notification commands. ``` - **Pass** — Code examples are available without interaction ```text llms.txt includes MCP config JSON and curl examples without requiring interaction. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text llms.txt states bearer key auth, OAuth grant by owner, and private beta access. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Homepage pricing section states $39/month per inbox, up to 10 agents, readable without interaction. ``` - **Pass** — Prices are stated, not gated ```text Price $39/month per inbox stated openly; beta users keep rate after launch. ``` - **Pass** — Pricing units and limits are explicit ```text Pricing unit is per inbox, up to 10 agents; billing starts when inbox provisioned. ``` - **Pass** — Agents identify pricing and its assumptions ```text 2 of 3 sessions were judged on pricing; 0 fell short. Kimi K3: Agent quotes $39/month per inbox covering up to 10 agents and calls out it's the early-bird private-beta rate with billing starting at inbox provisioning (seq 35), i.e. names the plan/tier assumptions behind the number. Qwen 3.8 Max: Final output states $39/month per inbox (up to 10 agents) and explicitly names it as the 'early-bird private-beta rate' with billing starting at inbox provisioning — figure paired with assumptions. This behavioural item does not affect the fast grade. ``` ### Activation - **Skipped** — An API reference or OpenAPI spec is reachable ```text llms.txt describes MCP tools but no API reference or OpenAPI spec was fetched. ``` - **Pass** — An MCP server is documented and well-formed ```text llms.txt documents MCP over HTTP with config JSON, tool table, and auth bearer key. ``` - **Skipped** — A CLI install path is documented ```text llms.txt states there is no comail SDK and documents no CLI install path. ``` - **Skipped** — SDK packages resolve on their registries ```text llms.txt explicitly says there is no comail SDK; no package registry result supplied. ``` - **Skipped** — Agent skills are published ```text No agent skills are mentioned or published in the fetched llms.txt. ``` [Full report data](https://www.ax-check.com/comail.io/report.json)