# AX Check: railway.com Checked 2026-10-12. Railway's quickstart, CLI, and pricing are all agent-ready out of the box. All 23 checked items passed: docs, CLI install, API/MCP references, and pricing tables are readable and actionable without any login wall. ## Onboarding needs a login ## Coding sessions All three independent sessions (DeepSeek V4.1 Flash, Kimi K3, Qwen 3.8 Max) completed the task and found pricing clearly laid out by plan tier, resource rates, and billing assumptions, with no gated numbers or guesswork required. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/62e1ec62-c9d6-49ec-8e23-19dce323b755) · [Read transcript](https://www.ax-check.com/railway.com/sessions/deepseek.json) Pricing was broken down by plan tier, resource rates, and billing rules, with clear assumptions about plan type, workload, and usage levels. Evidence: README.md pricing table lists Free/Trial/Hobby/Pro/Enterprise tiers with included usage, per-resource rates (RAM/CPU/egress/volume), and caveats like idle billing and non-rollover usage, sourced 'from Railway's current docs'. #### End-to-end onboarding - **Onboarding needs a login**: The sandbox had no Railway credentials (no RAILWAY_TOKEN, no CLI installed) and the agent cannot complete OAuth/dashboard signup without a human. It wrote a working script that calls the real hosted GraphQL endpoint and confirmed the call path is live by sending a dummy token and getting back a genuine 'Not Authorized' response from the real API, but never obtained or used a real, working credential, so no authenticated product operation (listing real projects, creating a resource) ever succeeded. Event 7: ```text RAILWAY-related env vars: (none) --- railway CLI: not found ``` Event 84: ```text Error: Not Authorized exit=0 ``` Event 86: ```text I got blocked on credentials. This environment has no RAILWAY_TOKEN, so I could not run the example against live data. You'll need a token from https://railway.com/account/tokens. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Railway API token available in the sandbox**: This is a missing-credentials/test-environment limitation, not a product defect: Railway requires a token (account, workspace, project, or OAuth) to call its Public API, and none was present or obtainable without a human visiting the dashboard to generate one. The agent correctly detected this, built a runnable script, and verified the request path hit the real endpoint and returned an authorization error, then stopped rather than faking success. Event 7: ```text RAILWAY-related env vars: (none) --- railway CLI: not found ``` Event 84: ```text Error: Not Authorized exit=0 ``` Event 86: ```text I got blocked on credentials. This environment has no RAILWAY_TOKEN, so I could not run the example against live data. ``` #### Suggested Changes - **Add an npm-installable first-party SDK reference to the API docs**: The agent found only a community-maintained 'railway' TypeScript SDK on npm and the official CLI package, with no first-party JS/TS SDK; this led it to hand-roll raw GraphQL fetch calls. Add a link to an official SDK (if one exists) in docs.railway.com/integrations/api, or clarify in that page that raw GraphQL + fetch is the recommended lightweight path, so developers don't waste time searching npm for an official package. Event 32: ```text railway - TypeScript SDK for Railway. @railway/urls - Generate links to Railway resources ``` Event 86: ```text Caveat: there's **no first-party JS/TS SDK on npm** — the official interfaces are the GraphQL API and `@railway/cli` (a community `railway` package exists). ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/4c720ea8-ced0-40ff-93fa-8ca93185a724) · [Read transcript](https://www.ax-check.com/railway.com/sessions/kimi.json) Pricing was given with the plan tiers, usage-metering basis, and workload assumptions spelled out, not just a bare number. Evidence: README.md and final output list Trial/Hobby/Pro tiers, per-vCPU/RAM/egress rates, and explicitly estimate 'fractional CPU, <512MB' usage for this example, while flagging rates may change. #### End-to-end onboarding - **Onboarding needs a login**: Agent could not obtain Railway credentials on its own. It probed the GraphQL API unauthenticated, got "Not Authorized", and explicitly stated account/token creation requires browser-based GitHub OAuth it cannot perform. It built deploy/teardown scripts but never ran them with a real token, so no authenticated operation against the hosted product was demonstrated. Event 15: ```text {"errors":[{"message":"Not Authorized","locations":[{"line":1,"column":3}],"path":["me"],"extensions":{"code":"INTERNAL_SERVER_ERROR"},"traceId":"1018431580856242284"}],"data":null} ``` Event 35: ```text Blocked: RAILWAY_TOKEN is not set. Create one at https://railway.com -> Account Settings -> Tokens, then re-run: RAILWAY_TOKEN=... node deploy.mjs exit=1 ``` Event 37: ```text I'm blocked on credentials.** The API rejects all calls without a token (`Not Authorized`, verified above), and creating an account/token requires browser-based GitHub login at railway.com → Account Settings → Tokens — which I can't do from here. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service way to mint a Railway API token**: Railway requires account creation and token generation through browser-based GitHub OAuth in the dashboard. This is normal product authentication behavior, not a defect, but it fully blocked the agent from running any authenticated deploy. The agent correctly identified this up front and built scripts that fail fast with clear next steps rather than faking success. Event 15: ```text Not Authorized ``` Event 35: ```text Blocked: RAILWAY_TOKEN is not set. ``` Event 37: ```text creating an account/token requires browser-based GitHub login at railway.com → Account Settings → Tokens — which I can't do from here ``` - **API-only deploys cannot push local source code**: The agent discovered that Railway's GraphQL API can create projects/services only from a prebuilt public image, not from pushed source, so server.js itself could not be deployed via API alone without connecting GitHub or publishing an image first. This is a product/workflow limitation the agent worked around by defaulting to a public test image (traefik/whoami). Event 37: ```text API-only deploys can't push source code, so `deploy.mjs` deploys a prebuilt public image (`traefik/whoami`, overridable via `IMAGE=`) ``` #### Suggested Changes - **Add a programmatic/non-browser token issuance path for automation and CI use cases**: Currently the only way to get a RAILWAY_TOKEN is via browser GitHub OAuth in the dashboard (Account Settings -> Tokens), which blocks any headless or agent-driven setup. Verify success by confirming a service account or CLI-based device-flow token can be created without opening a browser. Event 37: ```text creating an account/token requires browser-based GitHub login at railway.com → Account Settings → Tokens — which I can't do from here ``` - **Document the image-only limitation of API-only deploys in the GraphQL API reference**: The agent had to infer through trial that serviceCreate from the hosted API only supports a public image source, not pushed source code, requiring a GitHub link or published image to deploy custom code. Stating this constraint directly in the API docs for serviceCreate would save new integrators from discovering it by experimentation. Event 37: ```text API-only deploys can't push source code, so `deploy.mjs` deploys a prebuilt public image (`traefik/whoami`, overridable via `IMAGE=`). Deploying `server.js` itself means either connecting a GitHub repo or pushing your own image ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/59b6fe25-c598-490e-883c-01c0bd867f79) · [Read transcript](https://www.ax-check.com/railway.com/sessions/qwen.json) Pricing was given with explicit assumptions: plan tier, per-vCPU/memory billing, egress, and a note to verify current numbers. Evidence: Final output at seq 34 lists Hobby/Pro seat pricing, compute/memory rates, egress/volume extras, and flags it as approximate/verify-at-source. #### End-to-end onboarding - **Onboarding needs a login**: Agent checked for a Railway API token in the environment (none found), tested an unauthenticated GraphQL call to Railway's public API (got a generic 200 HTTP status but a 'Not Authorized' GraphQL error), then built local example files and stopped, explicitly asking the user to supply a RAILWAY_API_TOKEN. No credential was ever obtained by the agent itself, and no authenticated operation against the hosted Railway API was performed. Event 3: ```text env | grep -i -E 'railway|RAILWAY' ; ls ~/.railway 2>/dev/null; which railway; ls /sandbox/repo ``` Event 20: ```text {"errors":[{"message":"Not Authorized","locations":[{"line":1,"column":3}],"path":["projects"],"extensions":{"code":"INTERNAL_SERVER_ERROR"},"traceId":"8953477107760377050"}],"data":null} ``` Event 34: ```text Status: blocked on `RAILWAY_API_TOKEN`.** Send it over (or run `railway login` and share the token) and I'll deploy the example end-to-end and hand you the public URL. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Railway API token available to the agent**: The sandbox environment had no RAILWAY_API_TOKEN, no CLI session, and no config directory for Railway. This is a missing-credentials limitation of the test environment, not a product defect — Railway's API legitimately requires authentication for account-specific operations like listing projects. Event 3: ```text env | grep -i -E 'railway|RAILWAY' ; ls ~/.railway 2>/dev/null; which railway; ls /sandbox/repo ``` Event 20: ```text {"errors":[{"message":"Not Authorized","locations":[{"line":1,"column":3}],"path":["projects"],"extensions":{"code":"INTERNAL_SERVER_ERROR"},"traceId":"8953477107760377050"}],"data":null} ``` Event 34: ```text Status: blocked on `RAILWAY_API_TOKEN`.** Send it over (or run `railway login` and share the token) and I'll deploy the example end-to-end and hand you the public URL. ``` #### Suggested Changes - **Clarify the GraphQL error response for unauthenticated requests**: When an unauthenticated POST to https://backboard.railway.com/graphql/v2 queries the 'projects' field, the API returns an HTTP 200 status but a GraphQL-level 'Not Authorized' error tagged with extensions.code 'INTERNAL_SERVER_ERROR'. Consider returning a 401/403 HTTP status and a clearer extensions.code (e.g. UNAUTHENTICATED) so SDK/API consumers can distinguish auth failures from server-side bugs without needing docs lookup. Verify by repeating the same curl call with no token and confirming the HTTP status and error code reflect the authentication failure. Event 15: ```text 200 ``` Event 20: ```text {"errors":[{"message":"Not Authorized","locations":[{"line":1,"column":3}],"path":["projects"],"extensions":{"code":"INTERNAL_SERVER_ERROR"},"traceId":"8953477107760377050"}],"data":null} ``` ### Task given to each agent Help me build a simple example using Railway. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 100/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Pass** — Homepage answers Markdown requests ```text Homepage served text/markdown at /index.md with 200 and markdown content type. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs, API, MCP, skills, and machine-readable endpoints with descriptions. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt groups links under clear headings like For AI Agents and Machine-Readable Endpoints. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt names the remote MCP server, GraphQL API, OpenAPI, and agent skills. ``` - **Pass** — A compact guide representation exists ```text Homepage serves text/markdown at /index.md and /llms.txt; docs pages also return Markdown. ``` - **Pass** — A focused guide is directly retrievable ```text Quick-start and cloud-agents quickstart are directly retrievable as Markdown with concrete steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text Cloud-agents quickstart Markdown is 930 tokens, well under the 8000-token budget. ``` - **Pass** — Product-docs links survive format changes ```text Homepage Markdown retains docs, llms.txt, MCP, and template links; docs routes remain reachable. ``` - **Pass** — The compact guide is independently actionable ```text Quick-start gives concrete steps: railway init, railway up, railway open, plus GitHub and Docker paths. ``` - **Pass** — Install and next-step links resolve ```text Fetched install and next-step pages (quick-start, CLI, MCP, skills) all returned 200. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quick-start gives concrete GitHub, CLI, Docker, and template deploy steps. ``` - **Pass** — Installation commands are extractable ```text CLI page lists curl, Homebrew, npm, Scoop, and binary install commands. ``` - **Pass** — Code examples are available without interaction ```text Quick-start and CLI pages show inline commands like railway init and railway up. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text CLI page states railway login and RAILWAY_TOKEN/RAILWAY_API_TOKEN auth boundaries. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders full plan and usage tables as static Markdown, no interaction needed. ``` - **Pass** — Prices are stated, not gated ```text Plan fees and per-second usage rates stated openly; only Enterprise is contact-sales. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: $/vCPU/s, $/GB/s, per GB egress, plan limits per service. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Pricing was broken down by plan tier, resource rates, and billing rules, with clear assumptions about plan type, workload, and usage levels. Evidence: README.md pricing table lists Free/Trial/Hobby/Pro/Enterprise tiers with included usage, per-resource rates (RAM/CPU/egress/volume), and caveats like idle billing and non-rollover usage, sourced 'from Railway's current docs'. Kimi K3: Pricing was given with the plan tiers, usage-metering basis, and workload assumptions spelled out, not just a bare number. Evidence: README.md and final output list Trial/Hobby/Pro tiers, per-vCPU/RAM/egress rates, and explicitly estimate 'fractional CPU, <512MB' usage for this example, while flagging rates may change. Qwen 3.8 Max: Pricing was given with explicit assumptions: plan tier, per-vCPU/memory billing, egress, and a note to verify current numbers. Evidence: Final output at seq 34 lists Hobby/Pro seat pricing, compute/memory rates, egress/volume extras, and flags it as approximate/verify-at-source. This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text OpenAPI 3.1 spec at railway.com/openapi.json describes the GraphQL endpoint and OAuth flows. ``` - **Pass** — An MCP server is documented and well-formed ```text MCP server documented at mcp.railway.com with server card, tools, and OAuth/CLI setup. ``` - **Pass** — A CLI install path is documented ```text CLI install documented via curl, Homebrew, npm, Scoop, and pre-built binaries. ``` - **Pass** — SDK packages resolve on their registries ```text npm registry lookup for @railway/cli returned HTTP 200. ``` - **Pass** — Agent skills are published ```text Agent skills published: use-railway skill installable via railway skills or npx skills. ``` [Full report data](https://www.ax-check.com/railway.com/report.json)