# AX Check: smithery.ai Checked 2026-10-29. Docs and CLI install are clear, but the pricing page shows no prices 16 of 23 checks passed, including quickstart, CLI, and SDK install steps. The pricing page itself has no plan prices, units, or limits listed. ## End-to-end onboarding not demonstrated ## Coding sessions All three independent sessions (DeepSeek V4.1 Flash, Kimi K3, Qwen 3.8 Max) completed and reported tier prices, free-tier RPC quotas, and overage rates they found on the live pricing page, despite the page's rendered content lacking that detail. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/57b727b5-972b-43ef-b8a5-2160fd8ce034) · [Read transcript](https://www.ax-check.com/smithery.ai/sessions/deepseek.json) Gave tier prices pulled from the live pricing page, with their basis: monthly RPC quotas, namespace limits, $10 credits then $0.10 per 1K RPCs. Also noted that paid plans are being refunded. (seq 96, seq 119, seq 146) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No credentials were obtained or used. The agent found @smithery/api requires a SMITHERY_API_KEY env var it never had, and confirmed the hosted Connect host (smithery.run) does not resolve while api.smithery.ai returns 401 without a key. No authenticated operation was attempted or demonstrated; the only successful calls were anonymous reads of the public registry (api.smithery.ai/servers), which needs no credentials. Event 105: ```text Missing authentication token. Provide an API key or session token in the Authorization header. ``` Event 144: ```text === https://api.smithery.ai/servers === HTTP 200 size=7611 {"servers":[{"id":"50f26566-7dfe-4842-a5b8-1a9407fb91f4","qualifiedName":"brave" ``` Event 118: ```text If not provided, a new client will be created using the SMITHERY_API_KEY environment variable. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Smithery's connection-management platform shuts down in 2 days**: The live /future page states the Smithery API, Connect, Toolbox, accounts, API keys, CLI and SDKs stop working on October 31, 2026 (session ran Oct 29, 2026). This is a product lifecycle decision by Smithery, not agent error or missing test credentials — it makes building any new integration against the hosted Connect workflow pointless regardless of credentials. Event 128: ```text On October 31, 2026, we're shutting down Smithery's connection management services. ``` Event 128: ```text Server URLs on run.tools and server.smithery.ai ``` Event 128: ```text Smithery API, Connect, and Toolbox ``` Event 128: ```text The Smithery CLI and SDKs. We'll mark the current versions as deprecated. ``` - **Hosted Connect endpoint smithery.run has no working DNS**: The default connectBaseURL baked into @smithery/api ('https://smithery.run') failed to resolve, and api.smithery.ai rejects unauthenticated requests. This is a product/environment issue (likely tied to the announced shutdown) rather than an agent mistake — it blocks any createConnection() call regardless of API key. Event 110: ```text --- DNS smithery.run --- no DNS ``` Event 142: ```text smithery.run -> %{http_code}\n ``` Event 105: ```text === smithery.run upsert without key === [HTTP 000] ``` - **No self-service API key available for the SDK quickstart**: The documented quickstart (createConnection via @smithery/api) requires a SMITHERY_API_KEY, obtainable only from smithery.ai/account/api-keys behind a human login — the agent had no path to generate one in-session. This is a missing-credentials limitation, not a product defect. Event 55: ```text Get your API key from [smithery.ai/account/api-keys](https://smithery.ai/account/api-keys). ``` Event 118: ```text If not provided, a new client will be created using the SMITHERY_API_KEY environment variable. ``` #### Suggested Changes - **Update the pricing and docs pages to reflect the October 31 shutdown**: The main docs (smithery.ai/docs), SDK quickstarts, and pricing page (smithery.ai/pricing) still describe Connect, namespaces, and paid plans as active products with no shutdown notice, while only the homepage banner and /future page mention the October 31, 2026 retirement. Add the shutdown notice directly to /docs, /docs/use/connect, and /pricing so developers evaluating the SDK do not invest time in a workflow being decommissioned in days. Verify by checking that those pages render the same banner/callout seen on the homepage and /future. Event 64: ```text Smithery's MCP Proxy and managed identity shut down on October 31. See what's changing ``` Event 55: ```text Smithery lets you add MCP server integrations to your app without managing the complexity yourself ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/5a3c90ff-ebd6-4787-86a2-de16110a1123) · [Read transcript](https://www.ax-check.com/smithery.ai/sessions/kimi.json) Gave prices pulled from the live pricing page with their basis: Free gets 50K RPCs and 3 namespaces; $10/mo is credits, then $0.10 per 1K RPCs, with 100 namespaces; Custom tier quoted separately. Defined an RPC as one tool call. (seq 42, seq 47, seq 35, seq 144) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No Smithery credentials were obtained during the session; the agent checked the environment for an API key and found none, then built the example around the public unauthenticated registry endpoint. The authenticated SDK path (creating a connection and calling a hosted tool) was never executed against the real API — only inspected via local TypeScript type definitions. The session ends asking the human to supply an API key and namespace slug. Event 51: ```text env | grep -i smithery; echo "exit: $?" ``` Event 54: ```text No API key in the environment, so I'll build the example around Smithery's public registry API (which needs no key), and include an optional path that calls a hosted MCP server if a key is provided. ``` Event 144: ```text To go further, drop a key into the environment: `SMITHERY_API_KEY=sk-... SMITHERY_NAMESPACE=... node example.js` ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No API key available to exercise authenticated connection/tool-call path**: The sandbox environment had no SMITHERY_API_KEY, which is a normal credential gap rather than a product defect. The agent worked around it by using the public registry search endpoint and only validated the authenticated SDK calls (connections.create, tools.list) against local type definitions, never against the live API. Event 52: ```text exit: 1 ``` Event 64: ```text Error: The SMITHERY_API_KEY environment variable is missing or empty; either provide it, or instantiate the Smithery client with an apiKey option, like new Smithery({ apiKey: 'My API Key' }). ``` - **Registry API became unresponsive mid-session**: After multiple successful calls to registry.smithery.ai, requests began hanging and timing out (curl exit code 28, HTTP 000) for the remainder of the session, including after adding explicit fetch timeouts. This looks like a transient outage or rate limiting on Smithery's side, not an agent error, since earlier identical calls succeeded. Event 126: ```text HTTP 000 in 20.002818s HTTP 000 in 20.002978s Command exited with code 28 ``` Event 142: ```text The operation was aborted due to timeout ``` #### Suggested Changes - **Fix or document the intermittent registry.smithery.ai timeout**: Multiple identical GET requests to registry.smithery.ai/servers succeeded early in the session (HTTP 200) then began hanging past 30 seconds later in the same session, even with explicit client-side timeouts set. Product owner should check registry API uptime/rate-limit behavior around this window and confirm whether an unauthenticated client can get throttled; verify by re-running the same query pattern used here (GET /servers?q=&pageSize=5) repeatedly in a short span. Event 124: ```text https://registry.smithery.ai/servers?q=time&pageSize=5 ``` Event 132: ```text HTTP 000 in 30.002895s Searching Smithery registry for "exa"... ``` - **Add a quickstart code sample for the @smithery/api SDK's connection/tool-call flow**: The agent had to reverse-engineer the SDK's authenticated API surface by grepping compiled .d.ts files in node_modules (resources/connections/connections.d.ts, tools.d.ts) because no runnable example for client.connections.create + client.connections.tools.list was found. Adding a short, copyable snippet for this exact flow to the SDK README or smithery.ai docs would let a new developer skip this discovery step; verify by checking whether the docs page for the SDK shows a connections.create example. Event 100: ```text const connection = await client.connections.create(\n * 'namespace',\n * ); ``` Event 110: ```text Now I'll fix the keyed path in the example to match the real SDK surface: ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/a3a5a6b8-d4d4-4d9e-8ef6-16ed3394b298) · [Read transcript](https://www.ax-check.com/smithery.ai/sessions/qwen.json) Quoted Free/$10-mo/Custom tiers taken from the pricing page, with RPC metering, $10 credit then $0.10/1K overage, retrieval date and the shutdown/refund caveat (seq 47, seq 66, smithery-example/README.md) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: The agent never attempted to create a Smithery account or obtain an API key. It only called public, unauthenticated registry endpoints (/health, /servers, /servers/{name}), then deliberately probed the authenticated connect endpoint without a key to confirm it returns 401. No credential acquisition occurred, so no authenticated product operation was demonstrated — the only writes attempted were an intentionally-unauthenticated test call. Event 64: ```text connect API without an API key -> HTTP 401 {"error":"unauthorized","message":"Missing Authorization header"} ``` Event 48: ```text {"error":"unauthorized","message":"Missing Authorization header"} ``` #### Hallucinated URLs - **Guessed smithery.run connect host does not resolve**: The agent constructed a POST to https://smithery.run/demo based on docs examples of the Connect REST host pattern, but the host does not resolve at all (DNS failure), not just a 404. This came from inference about the product's documented host pattern rather than a copied working link, and the request failed outright. Event 40: ```text timeout 15 curl -sS -w "\n[%{http_code}]\n" -X POST "https://smithery.run/demo" -H "Content-Type: application/json" -d '{"mcpUrl":"https://mcp.exa.ai"}' ``` Event 42: ```text curl: (6) Could not resolve host: smithery.run ``` #### Blockers - **Smithery's managed connection product is shutting down in two days**: Product-level blocker, not agent error or missing credentials: Smithery's own site states the MCP Proxy, Connect API, accounts, API keys, CLI, and SDKs are shutting down on October 31, 2026 (session ran October 29, 2026). The agent verified this live — the Connect REST host no longer resolves and the proxy returns 401 — making any new integration built on the documented developer workflow (CLI login, create connection, call tool) non-durable or already broken. Event 30: ```text On October 31, 2026, we're shutting down Smithery's connection management services. Smithery will remain as a public (non-proxied) MCP registry. ``` Event 30: ```text What stops working on October 31 Server URLs on run.tools and server.smithery.ai Smithery API, Connect, and Toolbox Smithery accounts and API keys The Smithery CLI and SDKs. ``` Event 42: ```text curl: (6) Could not resolve host: smithery.run ``` Event 48: ```text {"error":"unauthorized","message":"Missing Authorization header"} ``` - **No Smithery API key available for the authenticated workflow**: Session limitation, not a product defect: the sandbox environment only exposed a PI_GATEWAY_API_KEY, no Smithery credentials, and there is no self-serve way shown in the transcript to mint one without human/browser login. This is an expected restriction of the test environment rather than friction in the product itself, and the agent correctly treated it as a stop condition rather than attempting a workaround. Event 5: ```text PI_GATEWAY_API_KEY= ``` Event 66: ```text I also have no Smithery account/API key and can't create one (and it'd be void in 2 days). ``` #### Suggested Changes - **Add a prominent shutdown banner date to the docs quickstart pages**: The CLI/SDK quickstart at smithery.ai/docs/use/connect.md (and llms.txt index) still presents 'smithery auth login', connection creation, and tool calls as the standard path with no mention of the October 31, 2026 shutdown; that context only appears on a separate /future page. Add the same shutdown notice shown on smithery.ai/future directly into the Connect quickstart doc so a new developer does not start building on a soon-to-be-dead workflow. Verify by loading smithery.ai/docs/use/connect.md and confirming the shutdown date appears before the Quick Start code block. Event 23: ```text # 1. Log in to Smithery smithery auth login # 2. Connect to the Exa search server smithery mcp add exa --id exa ``` Event 30: ```text On October 31, 2026, we're shutting down Smithery's connection management services. ``` - **Fix or remove the broken smithery.run host referenced in Connect docs**: The cURL example in the Connect quickstart (smithery.ai/docs/use/connect.md) shows POSTing to https://smithery.run/my-app, but that host does not resolve (DNS failure) as of October 29, 2026, two days before the stated shutdown. Update the example to the currently-live api.smithery.ai host or add a note that smithery.run is already decommissioned. Verify with a DNS/curl check against the documented host before republishing. Event 23: ```text curl -X POST "https://smithery.run/my-app" \ -H "Authorization: Bearer $SMITHERY_API_KEY" ``` Event 42: ```text curl: (6) Could not resolve host: smithery.run ``` ### Task given to each agent Help me build a simple example using Smithery. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: B · 68/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — llms.txt provides an actionable documentation index ```text /llms.txt returns HTTP 404; docs index lives at /docs/llms.txt instead. ``` - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown (1006 tokens) when requested with Accept: text/markdown. ``` - **Skipped** — llms.txt provides navigation guidance ```text No /llms.txt body exists (404), so its navigation guidance cannot be judged. ``` - **Skipped** — llms.txt mentions offered API, MCP, and skills ```text No /llms.txt body exists (404), so API/MCP/skills mentions cannot be judged. ``` - **Pass** — A compact guide representation exists ```text Homepage serves text/markdown (1006 tokens) and docs pages offer .md routes like connect.md. ``` - **Pass** — A focused guide is directly retrievable ```text connect.md is a directly retrievable Markdown guide with quick-start CLI steps and REST API details. ``` - **Pass** — Equivalent instructions fit a token budget ```text Homepage Markdown is 1006 tokens, well under the 8000-token budget and far below the HTML extraction. ``` - **Pass** — Product-docs links survive format changes ```text Homepage Markdown retains docs links (namespaces, connect, token-scoping) to the same documentation routes. ``` - **Pass** — The compact guide is independently actionable ```text Server pages give a concrete npx smithery quickstart: add, tool list, tool get, tool call. ``` - **Pass** — Install and next-step links resolve ```text Fetched install/next-step routes (docs connect, server pages, skills) all returned 200. ``` ### Onboarding - **Skipped** — Docs lead to a relevant quickstart ```text No fetched Smithery docs page; only a 404 llms.txt and a CLI README were retrieved. ``` - **Pass** — Installation commands are extractable ```text CLI README gives install commands: npx skills add smithery/cli and npm install -g smithery@latest. ``` - **Pass** — Code examples are available without interaction ```text README shows inline command examples for mcp search, add, tool call, and publish. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text README states Node.js 20+ requirement and documents smithery auth login/token flows. ``` ### Pricing - **Failed** — Pricing is readable without interaction ```text Smithery's /pricing page returned only nav and footer, no plan or price content. ``` - **Failed** — Prices are stated, not gated ```text No Smithery plan prices are stated on the fetched pricing page. ``` - **Failed** — Pricing units and limits are explicit ```text Pricing page shows no units, quotas or usage limits for Smithery plans. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Gave tier prices pulled from the live pricing page, with their basis: monthly RPC quotas, namespace limits, $10 credits then $0.10 per 1K RPCs. Also noted that paid plans are being refunded. (seq 96, seq 119, seq 146) Kimi K3: Gave prices pulled from the live pricing page with their basis: Free gets 50K RPCs and 3 namespaces; $10/mo is credits, then $0.10 per 1K RPCs, with 100 namespaces; Custom tier quoted separately. Defined an RPC as one tool call. (seq 42, seq 47, seq 35, seq 144) Qwen 3.8 Max: Quoted Free/$10-mo/Custom tiers taken from the pricing page, with RPC metering, $10 credit then $0.10/1K overage, retrieval date and the shutdown/refund caveat (seq 47, seq 66, smithery-example/README.md) This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text API reference pages embed OpenAPI specs for Smithery Platform API endpoints. ``` - **Pass** — An MCP server is documented and well-formed ```text Server pages document MCP servers with quick start, endpoint, and tool lists. ``` - **Pass** — A CLI install path is documented ```text CLI install documented: npm install -g smithery@latest and npx smithery usage. ``` - **Pass** — SDK packages resolve on their registries ```text npm registry lookups for smithery, @smithery/cli, and @smithery/api returned HTTP 200. ``` - **Pass** — Agent skills are published ```text Skills registry lists 25,352 skills including smithery-ai/cli with install counts. ``` [Full report data](https://www.ax-check.com/smithery.ai/report.json)