# AX Check: fly.io Checked 2026-10-20. Fly.io quickstart and pricing are usable with no login wall. 21 of 23 checklist items passed: docs, CLI install, and pricing tables are all readable without interaction. One session's own free-credit math didn't add up. ## Onboarding needs a login ## Coding sessions All three independent sessions completed the task, pulling concrete pricing straight from Fly's docs with stated machine sizes, regions, and usage assumptions, rather than hitting a paywall or login wall. One session's reasoning about free-tier credit contradicted itself, but prices were otherwise clearly sourced and explained across all three runs. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/58f76378-0488-4fb2-93f2-62c10139a52d) · [Read transcript](https://www.ax-check.com/fly.io/sessions/deepseek.json) Prices came from the fetched Fly docs and named their basis: machine size (shared-cpu-1x 256 MB is $2.19/mo running continuously), traffic region, an app plus Postgres example (~$44) and regional variance. (seq 26, seq 97) #### End-to-end onboarding - **Onboarding needs a login**: Agent never obtained real Fly.io credentials. It probed the Machines API unauthenticated (confirming a 401 'token validation error'), built a local script requiring a human-supplied FLY_API_TOKEN, and explicitly stated it has no token, no flyctl, and no Fly account in the sandbox. No authenticated operation against the hosted API was ever performed. Event 35: ```text unauth POST /v1/apps -> 401 {"error":"Authenticate: token validation error"} ``` Event 97: ```text Where I'm blocked:** I have no `FLY_API_TOKEN`, no `flyctl`, and no Fly account in this sandbox. ``` Event 97: ```text Give me a token/org and I can run and manage Option 1 directly. ``` #### Hallucinated URLs - **Guessed OpenAPI spec locations on api.machines.dev**: Agent tried several plausible but invented spec URLs directly on the API host before finding the real spec at docs.fly.io; these guesses returned 404 but are not evidence of a broken documented link, just speculative path guessing that failed. Event 46: ```text https://api.machines.dev/openapi.json -> 404 19 https://api.machines.dev/swagger.json -> 404 19 https://api.machines.dev/v1/openapi.json -> 404 19 ``` #### Blockers - **No Fly.io account or API token available in sandbox**: The sandbox had no FLY_API_TOKEN, no flyctl installation, and no pre-authenticated Fly config. This is a test-environment/credential limitation, not a product defect, since Fly.io requires normal account creation and login (interactive 'fly auth login') to obtain a token, which the agent correctly declined to attempt given the session's no-login/no-long-running-command constraints. Event 13: ```text no ~/.fly config ``` Event 97: ```text I have no `FLY_API_TOKEN`, no `flyctl`, and no Fly account in this sandbox. Fly requires an account with a payment method, and the standard quickstart (`fly auth login` → `fly launch`/`fly deploy`) is interactive **and** long-running ``` #### Suggested Changes - **Publish a direct link to the Machines API OpenAPI spec from the main pricing/quickstart docs**: The agent had to guess several URLs on api.machines.dev (openapi.json, swagger.json, v1/openapi.json) before finding the real spec at docs.fly.io/api/machines/openapi.json via a page scrape. Adding a direct, discoverable link to this spec on the main Fly.io docs or pricing page would save this kind of trial-and-error. Verify by confirming a new session finds the spec URL without probing api.machines.dev directly for guessed filenames. Event 46: ```text https://api.machines.dev/openapi.json -> 404 19 https://api.machines.dev/swagger.json -> 404 19 https://api.machines.dev/v1/openapi.json -> 404 19 https://docs.machines.dev/openapi.json -> 200 298860 ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/6486b47f-985f-43c8-a6ac-c135bc16ba18) · [Read transcript](https://www.ax-check.com/fly.io/sessions/kimi.json) Gave figures with a basis: shared-cpu-1x at 256MB for about $1.94/mo, under $3/mo for the 24/7 demo, and NA/EU bandwidth. But its ~$5/mo free credit contradicts its own reasoning that the free tier ended, and none of it was checked. (seq 11, seq 12) #### End-to-end onboarding - **Onboarding needs a login**: Agent never obtained Fly.io credentials. No flyctl binary installed, no FLY_API_TOKEN present in environment, and the agent explicitly states it cannot complete signup (email/GitHub OAuth plus payment method) on its own. It wrote a deploy script but never executed it against the live Machines API, so no authenticated operation occurred. Event 4: ```text /bin/bash: line 1: flyctl: command not found ``` Event 12: ```text Fly.io's developer workflow — both the `flyctl` CLI and the hosted Machines REST API — requires an authenticated account. Sign-up involves email/GitHub OAuth plus adding a payment method, which I can't do for you, and there's no `FLY_API_TOKEN` in this environment. So **I can't deploy anything myself right now**. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Fly.io credentials available to agent**: Fly.io requires account signup (email/GitHub OAuth) plus a payment method before any CLI or API deployment can occur. The sandbox had no flyctl binary and no FLY_API_TOKEN environment variable, so the agent could not self-serve a credential. This is an expected product requirement (payment-gated deploys) rather than a defect, but it fully blocked the deployment task in this session. Event 4: ```text /bin/bash: line 1: flyctl: command not found ``` Event 12: ```text Sign-up involves email/GitHub OAuth plus adding a payment method, which I can't do for you, and there's no `FLY_API_TOKEN` in this environment. ``` #### Suggested Changes - **Offer a token-free sandbox or trial credential for the Machines API**: The agent could write a complete, correct deploy script against https://api.machines.dev/v1 but had no way to test it without a paid account and manually generated token. Providing a short-lived trial token or sandbox org (reachable via a documented self-service endpoint) would let automated agents or new developers validate the quickstart end-to-end. Verify by having a fresh, credential-less environment successfully create and reach an app via the Machines API without human-provided secrets. Event 12: ```text If you generate a token (`flyctl tokens create deploy` after `flyctl auth signup/login`, or from the dashboard) and set it as `FLY_API_TOKEN`, everything below will run as-is. This is a hard stop for actual deployment ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/f080fbe4-7890-4cbd-bb4b-225a754cd785) · [Read transcript](https://www.ax-check.com/fly.io/sessions/qwen.json) Prices from the fetched pricing page, with stated basis: iad/ewr regions, 30-day always-on, shared-cpu-1x/256MB at $2.19, auto-stop idle cost, and Postgres Basic behind ~$44. (seq 26, seq 75, flyio-example/README.md) #### End-to-end onboarding - **Onboarding needs a login**: Agent installed flyctl and confirmed network reachability to the Fly API, but never obtained real Fly.io credentials. `fly auth whoami` failed because no token existed, `fly auth login` requires an interactive browser OAuth flow the agent cannot complete, and `FLY_API_TOKEN` was never set. A POST to the live Machines API correctly returned 401 (proving the API is real but unauthenticated). No authenticated operation (app/machine creation) was ever performed — the deploy script was written and syntax-checked only, never executed against the API. Event 21: ```text Error: no access token available. Please login with 'flyctl auth login' ``` Event 21: ```text FLY_API_TOKEN not set ``` Event 65: ```text POST /v1/apps -> 401 ``` Event 54: ```text server.js parses: true scripts OK ``` #### Hallucinated URLs - **Guessed Fly.io docs pricing/API paths that 404'd**: Agent tried several plausible but invented documentation URLs that returned 404, then fell back to the real pricing page and llms.txt index. These look like guesses rather than demonstrated product defects since the agent immediately found working equivalents. Event 22: ```text https://fly.io/docs/machines/machines-price/ -> 404 https://fly.io/docs/reference/pricing/ -> 404 ``` Event 35: ```text 404 404 ``` #### Blockers - **No self-service way to obtain a Fly.io API token**: The sandbox had no pre-provisioned Fly credentials. `fly auth login` requires an interactive browser-based OAuth flow that a headless agent cannot drive, and there was no FLY_API_TOKEN in the environment. This is a normal authentication requirement of the product, not a product defect, but it fully blocked any authenticated deploy in this session. Event 21: ```text Error: no access token available. Please login with 'flyctl auth login' ``` Event 21: ```text FLY_API_TOKEN not set ``` Event 75: ```text fly auth login` is an interactive browser OAuth flow I can't complete. Give me a token (`fly tokens create deploy -a pi-fly-example`) and `scripts/deploy.sh` runs end to end. ``` #### Suggested Changes - **Fix or redirect dead pricing/API doc paths**: Requests to https://fly.io/docs/machines/machines-price/, https://fly.io/docs/reference/pricing/, and https://fly.io/docs/machines/api/machines-api/ all returned 404 while the agent was hunting for pricing and Machines API docs. Add redirects from these paths to the live pages (https://fly.io/docs/about/pricing/ and the docs.fly.io/api/machines/* set) so agents and users following plausible guessed links land on real content; verify by curling the old paths and confirming a 200 or 3xx instead of 404. Event 22: ```text https://fly.io/docs/machines/machines-price/ -> 404 https://fly.io/docs/reference/pricing/ -> 404 ``` Event 35: ```text 404 404 ``` ### Task given to each agent Help me build a simple example using Fly.io. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 100/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown with 200 when requested with Markdown Accept header. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs index, full text, product pages, SDKs, CLI, and MCP server. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt groups content under Product, Documentation, SDKs, and Pages headings with guidance. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt lists Machines and Sprites OpenAPI specs, fly-go SDK, and Sprites MCP server. ``` - **Pass** — A compact guide representation exists ```text Homepage serves text/markdown; llms.txt states every page has a .md version. ``` - **Pass** — A focused guide is directly retrievable ```text Quickstart launch page retrievable as Markdown with concrete flyctl steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text Markdown quickstart is 608 tokens, well under the 8000-token budget. ``` - **Pass** — Product-docs links survive format changes ```text Homepage Markdown preserves docs, pricing, and getting-started links. ``` - **Pass** — The compact guide is independently actionable ```text Quickstart gives concrete steps: install flyctl, fly auth signup, fly launch, fly deploy, then next steps. ``` - **Pass** — Install and next-step links resolve ```text Install and next-step links resolve: flyctl install, launch, sign-up pages all fetched successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quickstart 'Launch your app' gives concrete steps: install flyctl, sign up, run fly launch. ``` - **Pass** — Installation commands are extractable ```text Install page lists extractable commands: brew install flyctl, curl install.sh, PowerShell install.ps1. ``` - **Pass** — Code examples are available without interaction ```text Quickstart and install pages show inline code commands without requiring interaction. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Sign-up page states account creation via fly auth signup and credit card requirement. ``` ### Pricing - **Failed** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 1 fell short. DeepSeek V4.1 Flash: Prices came from the fetched Fly docs and named their basis: machine size (shared-cpu-1x 256 MB is $2.19/mo running continuously), traffic region, an app plus Postgres example (~$44) and regional variance. (seq 26, seq 97) Kimi K3: Gave figures with a basis: shared-cpu-1x at 256MB for about $1.94/mo, under $3/mo for the 24/7 demo, and NA/EU bandwidth. But its ~$5/mo free credit contradicts its own reasoning that the free tier ended, and none of it was checked. (seq 11, seq 12) Qwen 3.8 Max: Prices from the fetched pricing page, with stated basis: iad/ewr regions, 30-day always-on, shared-cpu-1x/256MB at $2.19, auto-stop idle cost, and Postgres Basic behind ~$44. (seq 26, seq 75, flyio-example/README.md) This behavioural item does not affect the fast grade. ``` - **Pass** — Pricing is readable without interaction ```text Fly.io pricing.md renders full price tables as static Markdown, no interaction needed. ``` - **Pass** — Prices are stated, not gated ```text Concrete USD prices stated for Machines, Sprites, Postgres, support tiers. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: per-second, per GB/month, per CPU-hour, region multipliers. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Machines API reference and OpenAPI 3.1 spec reachable; Sprites API overview documents bearer auth. ``` - **Pass** — An MCP server is documented and well-formed ```text Sprites remote MCP server documented: endpoint, Streamable HTTP transport, OAuth 2.1 scopes, 24 tools. ``` - **Pass** — A CLI install path is documented ```text flyctl install documented for macOS, Linux, Windows via brew, curl script, PowerShell. ``` - **Pass** — SDK packages resolve on their registries ```text npm @fly/sprites and PyPI sprites-py registry lookups returned HTTP 200. ``` - **Skipped** — Agent skills are published ```text No agent skills published by Fly.io found in fetched evidence. ``` [Full report data](https://www.ax-check.com/fly.io/report.json)