# AX Check: semaphore.io Checked 2026-10-22. Docs and pricing are clear, but the llms.txt index skips them Pricing, quickstart and CLI install steps are all readable without login, but the llms.txt file lists only blog and marketing pages, omitting docs, API and MCP links. ## Onboarding needs a login ## Coding sessions All three independent sessions (DeepSeek V4.1 Flash, Kimi K3, Qwen 3.8 Max) completed and found pricing from the public pricing page: per-minute compute rates by machine size and OS, the $15/month free credit, and the 20-job concurrency limit, with two sessions producing per-run cost estimates. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/cb55ad32-f554-4cc4-ad4a-a65e9e434615) · [Read transcript](https://www.ax-check.com/semaphore.io/sessions/deepseek.json) Gave per-minute rates tied to machine size/OS, the $15/mo free-credit limit (about 2,000 2-vCPU minutes) and storage overage, all taken from the scraped pricing page; no cost estimate for the example workload (seq 99, seq 101) #### End-to-end onboarding - **Onboarding needs a login**: The agent never obtained a Semaphore API token, organization URL, or project ID, and performed no authenticated operation against the hosted product. It only confirmed the API requires auth (401 on an unauthenticated GET) and built/validated an example locally with no credentials. It explicitly stated it could not execute anything against a real account and stopped there, citing the missing API token/org URL/project ID as the blocker. Event 74: ```text == unauthenticated GET (expect 401) == HTTP 401 ``` Event 92: ```text error: Set SEMAPHORE_ORG_URL and SEMAPHORE_API_TOKEN first. exit=1 ``` Event 101: ```text Live runs need a personal **API token**, your **org URL**, and a **project ID** — none are available in this environment. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No API token or org URL available to authenticate**: The hosted Semaphore API requires a personal API token and organization URL, which are normally obtained via human signup (browser-based) and account settings. This is a credentials gap inherent to the sandboxed test environment, not a product defect — the agent correctly identified this as an unresolvable session limitation rather than a bug, and avoided running destructive or fabricated actions against the real account. Event 101: ```text Live runs need a personal **API token**, your **org URL**, and a **project ID** — none are available in this environment. ``` Event 92: ```text error: Set SEMAPHORE_ORG_URL and SEMAPHORE_API_TOKEN first. exit=1 ``` #### Suggested Changes None identified in this transcript. ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/6c9f122f-5dd5-4dca-b9c0-38c1c7aa11bf) · [Read transcript](https://www.ax-check.com/semaphore.io/sessions/kimi.json) Quoted per-minute rates from the fetched pricing page by OS and vCPU, plus $15/month in free credits; tied the $0.01–0.02 per-run estimate to the 2 vCPU x64 machine (seq 17, seq 31, .semaphore/semaphore.yml) #### End-to-end onboarding - **Onboarding needs a login**: Agent never obtained Semaphore credentials or performed any authenticated operation against the hosted product. It fetched the public pricing page and built a local example (Python script + unit test + .semaphore/semaphore.yml) but explicitly stated it could not sign up or call the API because account creation requires interactive GitHub/Bitbucket OAuth in a browser, which it could not do without human intervention. Event 31: ```text I could not execute a live pipeline run. Semaphore's hosted API requires an account, and account creation is interactive GitHub/Bitbucket OAuth in a browser — I have no credentials or org token in this environment, so there's no API endpoint I can authenticate to. ``` Event 29: ```text Ran 1 test in 0.000s OK Hello, Semaphore! ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service signup path for Semaphore account/API token**: Product behavior plus environment constraint: creating a Semaphore account requires interactive OAuth via GitHub/Bitbucket in a browser. The sandbox had no credentials, org token, or browser, so the agent could not reach any authenticated API surface to run a real pipeline. Event 31: ```text account creation is interactive GitHub/Bitbucket OAuth in a browser — I have no credentials or org token in this environment, so there's no API endpoint I can authenticate to ``` - **Self-hosted Community Edition requires a local service stack**: The open-source alternative to the hosted SaaS would require standing up a local server stack, which the task instructions explicitly forbade, so the agent correctly declined that path instead of attempting it. Event 31: ```text The alternative (Semaphore's open-source Community Edition) requires standing up a local server stack, which per your instructions I won't do. ``` #### Suggested Changes - **Add a token-based or CLI-only signup option for headless environments**: On the pricing/start-free flow at semaphoreci.com, account creation currently forces GitHub/Bitbucket OAuth in a browser. Providing a CLI- or API-key-based signup path (e.g., email + API token issuance without OAuth redirect) would let automated agents or CI scripts self-provision and verify by calling the REST API or sem CLI directly, which this session could not do. Event 31: ```text account creation is interactive GitHub/Bitbucket OAuth in a browser — I have no credentials or org token in this environment, so there's no API endpoint I can authenticate to ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/d2d012eb-07da-46f7-9f44-c4e1985bd46b) · [Read transcript](https://www.ax-check.com/semaphore.io/sessions/qwen.json) Quoted per-minute rates by machine size and OS from the fetched pricing page, plus the $15/month free credit and the 20-job concurrency limit. It estimated about $0.015 per run, assuming 2 jobs of about 1 minute each on f1-standard-2. (seq 45, seq 208) #### End-to-end onboarding - **Onboarding needs a login**: The agent correctly avoided local stacks and used the hosted Semaphore Cloud API/CLI, but every real operation requires a Semaphore account token, and token issuance requires a human to open a browser and approve a device-flow code. The agent started that flow twice and both times it was still waiting for human approval when the session ended, with no token ever written to the credential file. No authenticated product operation was ever demonstrated — all work product-side was limited to 401 probes and local file writes/tests. Event 86: ```text Open https://id.semaphoreci.com/device in a browser and enter the code. Sign in there (or create your account), then approve. Waiting... ``` Event 98: ```text --- sem.yaml -rw------- 1 root root 0 Oct 22 11:31 /sandbox/.sem.yaml ``` Event 202: ```text First, copy your one-time code: TRHR-RLCG Open https://id.semaphoreci.com/device in a browser and enter the code. Sign in there (or create your account), then approve. Waiting... ``` Event 62: ```text == orgs create, no auth HTTP:401 ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **Account creation requires a human-approved browser device flow**: No Semaphore API token or account could be obtained without a human clicking through a browser. This is a normal login/signup requirement (not a product defect) but it hard-blocked any authenticated API, MCP, or CLI call for the rest of the session. Event 86: ```text Open https://id.semaphoreci.com/device in a browser and enter the code. Sign in there (or create your account), then approve. Waiting... ``` Event 147: ```text signin: STILL WAITING for browser approval ``` - **Semaphore Cloud projects require a hosted Git remote**: Semaphore Cloud builds from a connected GitHub/Bitbucket repo. The sandbox repo had no remote and no GitHub credentials or gh CLI were available, so sem-ai project create could not be run even after auth. This is an environment/test-setup limitation, not a product bug. Event 92: ```text --- remotes --- branches * main ``` Event 98: ```text --- gh cli no gh ``` - **Agent error: --org flag used without required --org-host**: The agent's own CLI invocation was malformed on the first retry of the signin flow, producing a structured error instead of a working device code. This was a self-inflicted mistake the agent then corrected by reading the error message, not a product defect. Event 197: ```text "message": "--org requires --org-host: the create API does not return the new org's host, so it must be given explicitly" ``` #### Suggested Changes - **Fix the broken docs.semaphoreci.com API reference links**: docs.semaphoreci.com/api/, /api/overview/, and /api/projects/ all 404 and redirect structure has moved content to docs.semaphore.io. Update or redirect these legacy paths (or add redirects at the docs.semaphoreci.com host level) so old bookmarks/links land on the current reference/api page; verify by curling the old URLs and confirming a 200 or a working redirect instead of a Docusaurus 404. Event 18: ```text Page Not Found We could not find what you were looking for. Please contact the owner of the site that linked you to the original URL and let them know their link is broken. ``` - **Fix or remove the broken testbox docs link**: docs.semaphore.io/using-semaphore/ai/testbox returns a 404 Page Not Found despite testbox being referenced on the marketing homepage. Add the missing testbox documentation page or correct the link that points to it, then verify by loading the URL directly. Event 94: ```text #### https://docs.semaphore.io/using-semaphore/ai/testbox Semaphore Skip to main content Page Not Found We could not find what you were looking for. ``` - **Align the sem-ai-cli docs example with the canonical test-results publish syntax**: The sem-ai-cli reference page's epilogue example used a nonstandard test-results publish --name pytest report.xml || true pattern, while the test-reports page documents the canonical [[ -f report.xml ]] && test-results publish report.xml form. Reconcile the two docs pages so there is one documented epilogue pattern; verify by searching both pages for 'test-results publish' and confirming identical syntax. Event 164: ```text Type the publish command under epilogue.always.commands .semaphore/semaphore.yml # ... global_job_config : epilogue : always : commands : - '[[ -f report.xml ]] && test-results publish report.xml' ``` ### Task given to each agent Help me build a simple example using Semaphore. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: C · 63/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a text/markdown request; no Markdown representation offered. ``` - **Failed** — llms.txt provides an actionable documentation index ```text llms.txt exists but lists only blog, marketing and customer pages, no docs, API, CLI or quickstart links. ``` - **Failed** — llms.txt provides navigation guidance ```text llms.txt groups pages by marketing type (Pages, Posts, Customers) with no starting guidance for developers. ``` - **Failed** — llms.txt mentions offered API, MCP, and skills ```text llms.txt omits the offered MCP server, API reference and sem-ai CLI surfaces. ``` - **Pass** — A compact guide representation exists ```text Raw Markdown MCP guide served at raw.githubusercontent.com; llms.txt also published as Markdown. ``` - **Pass** — A focused guide is directly retrievable ```text MCP server guide fetched directly as plain Markdown with setup and capabilities. ``` - **Pass** — Equivalent instructions fit a token budget ```text MCP guide Markdown is 2732 tokens, well under the 8000-token budget. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be measured. ``` - **Pass** — The compact guide is independently actionable ```text Quickstart gives concrete steps: sign up, first project, build, deploy with linked tutorials. ``` - **Pass** — Install and next-step links resolve ```text Fetched quickstart, install, MCP and API pages all returned HTTP 200. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs quickstart page lists concrete steps: sign up, Hello World, build, deploy. ``` - **Pass** — Installation commands are extractable ```text CLI install commands shown: curl get.sh | bash and brew install semaphoreci/tap/sem. ``` - **Pass** — Code examples are available without interaction ```text Code examples appear inline in docs, e.g. sem-ai install, connect, and CLI commands. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Auth explicit: API token from me.semaphoreci.com/account, then sem connect. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders all compute rates, add-ons and support tiers as static text. ``` - **Pass** — Prices are stated, not gated ```text Prices stated openly: $0.003/min ARM, $15 free credits, add-on tiers listed. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: per-minute vCPU rates, 20 concurrent jobs, 20GB egress, 9.6GB cache. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Gave per-minute rates tied to machine size/OS, the $15/mo free-credit limit (about 2,000 2-vCPU minutes) and storage overage, all taken from the scraped pricing page; no cost estimate for the example workload (seq 99, seq 101) Kimi K3: Quoted per-minute rates from the fetched pricing page by OS and vCPU, plus $15/month in free credits; tied the $0.01–0.02 per-run estimate to the 2 vCPU x64 machine (seq 17, seq 31, .semaphore/semaphore.yml) Qwen 3.8 Max: Quoted per-minute rates by machine size and OS from the fetched pricing page, plus the $15/month free credit and the 20-job concurrency limit. It estimated about $0.015 per run, assuming 2 jobs of about 1 minute each on f1-standard-2. (seq 45, seq 208) This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Semaphore API reference documents v1alpha endpoints, auth, and curl examples. ``` - **Pass** — An MCP server is documented and well-formed ```text MCP Server page documents endpoint, OAuth/API-token auth, tools, and client setup. ``` - **Pass** — A CLI install path is documented ```text CLI install documented via curl script and Homebrew, plus sem-ai installer. ``` - **Skipped** — SDK packages resolve on their registries ```text No registry lookup result for a Semaphore SDK or packaged CLI was supplied. ``` - **Pass** — Agent skills are published ```text sem-ai plugin publishes Agent Skills; canonical inventory in sem-ai repo skills directory. ``` [Full report data](https://www.ax-check.com/semaphore.io/report.json)