# AX Check: github.com Checked 2026-10-06. GitHub's docs, CLI setup and pricing are all clear upfront 20 of 23 checklist items passed: CLI install, quickstart steps, API/MCP docs and pricing all render directly. Only the homepage's plain-text format drew attention. ## Onboarding needs a login ## Coding sessions All three independent sessions completed the task and reported pricing consistently: a free tier plus paid plans around $4 and $21 per user per month, with metered extras for Actions/Codespaces usage, each flagging the figures as worth double-checking against the live pricing page. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/ef4d719d-7add-46a4-9587-a6e0dcd9ab23) · [Read transcript](https://www.ax-check.com/github.com/sessions/deepseek.json) Final output gives seat pricing (Free/Pro ~$4/Team ~$4/Enterprise ~$21) plus metered Actions/Codespaces rates, explicitly caveated as approximate and checked against docs/pricing page (seq 29 curl to github.com/pricing). #### End-to-end onboarding - **Onboarding needs a login**: Agent found no GITHUB_TOKEN/GH_TOKEN/GITHUB_PAT in the environment and no gh CLI installed. It ran only unauthenticated GET requests against the public GitHub REST API (repo info, issues, rate_limit) — no sign-up, token creation, or authenticated write operation occurred. Agent explicitly declined to fabricate a token and stated a real token would be needed to demonstrate any mutating (authenticated) operation. Event 8: ```text GITHUB_TOKEN: unset GH_TOKEN: unset GITHUB_PAT: unset --- gh auth --- /bin/bash: line 1: gh: command not found ``` Event 25: ```text rate limit: 58/60 remaining ``` Event 37: ```text To demonstrate anything *mutating* (open an issue, comment on a PR, create a repo), I'd need a `GITHUB_TOKEN` with the relevant scopes. I won't fabricate one — that's the single thing I'd need from you to extend the example. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No GitHub token available for authenticated/write operations**: The sandbox had no GITHUB_TOKEN, GH_TOKEN, GITHUB_PAT, or gh CLI, so the agent could only call the public, unauthenticated GitHub REST API (60 requests/hour, read-only). This is a test-environment/missing-credentials limitation rather than a product defect — the agent correctly avoided fabricating a token and stopped short of any mutating operation. Event 8: ```text GITHUB_TOKEN: unset GH_TOKEN: unset GITHUB_PAT: unset --- gh auth --- /bin/bash: line 1: gh: command not found ``` Event 37: ```text there's no `gh` CLI and no token in this environment, so I'm on the unauthenticated tier — **public data only, 60 requests/hour** ``` #### Suggested Changes None identified in this transcript. ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/7a1a020b-bc52-4856-b618-c4b65c332097) · [Read transcript](https://www.ax-check.com/github.com/sessions/kimi.json) Final output states API access is free, breaks down rate-limit tiers (60/5,000/15,000 req-hr) and paid plan costs (~$4 and ~$21/user/mo), with explicit assumption that none of this example's usage triggers metered Actions/Packages/Codespaces costs. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No GitHub credentials (gh CLI, GITHUB_TOKEN) were present or obtained during the session. The agent used GitHub's public REST API unauthenticated and only performed read-only GET requests against public repo data. No authenticated write/create operation was attempted or demonstrated, and the agent explicitly flagged the lack of credentials as a limiting factor for write workflows. Event 17: ```text No credentials are configured (no `gh` CLI, no `GITHUB_TOKEN`), so I'll use GitHub's public REST API unauthenticated — fine for a lightweight read-only example (60 requests/hour limit). ``` Event 25: ```text Repo: psf/requests Stars: 54,492 Open issues:243 ``` Event 27: ```text no credentials are configured** (no token, no `gh` CLI), so right now I'm limited to read-only operations on public repos at 60 req/hr. Give me a `GITHUB_TOKEN` and I can do full read/write workflows. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No GitHub token available for write operations**: The sandbox had no gh CLI and no GITHUB_TOKEN environment variable, so the agent could only call the unauthenticated public REST API (60 requests/hour, read-only). This is a session/environment limitation (missing credentials) rather than a product defect — GitHub's API does support authenticated write workflows, they simply weren't accessible here without a human-supplied token. Event 17: ```text No credentials are configured (no `gh` CLI, no `GITHUB_TOKEN`), so I'll use GitHub's public REST API unauthenticated — fine for a lightweight read-only example (60 requests/hour limit). ``` Event 27: ```text no credentials are configured** (no token, no `gh` CLI), so right now I'm limited to read-only operations on public repos at 60 req/hr. Give me a `GITHUB_TOKEN` and I can do full read/write workflows. ``` #### Suggested Changes None identified in this transcript. ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/3e776fd0-211c-4871-acb1-ff2684259200) · [Read transcript](https://www.ax-check.com/github.com/sessions/qwen.json) Final output breaks down API rate-limit tiers (unauthenticated vs token) plus per-user/month plan pricing (Free/Pro ~$4/Team/Enterprise ~$21, Copilot tiers), explicitly flagged as approximate/verify-at-github.com/pricing. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: The agent only exercised GitHub's public, unauthenticated REST API (reads on octocat/Hello-World and facebook/react at 60 req/hour). No GITHUB_TOKEN was ever present in the environment, so no authenticated operation occurred. The agent even rewrote its own script to stop calling octokit.auth() because that check was failing, then hardcoded authed based on just the presence of an env var that was never set. The final summary's claim of 'authenticated (5k req/hour)' on the facebook/react run is contradicted by the later anonymous run and the absence of any token in env. Event 4: ```text env | grep -i -E 'gh_|github|token' | sed 's/=.*/=/'; which gh git node python3; gh --version 2>/dev/null ``` Event 36: ```text auth: anonymous (60 req/hour) ``` Event 38: ```text **No `GITHUB_TOKEN` in this environment**, so I could only exercise read-only endpoints. The `--star` write path is written and guarded but untested against a real token ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No GitHub token available for authenticated/write operations**: The sandbox environment had no GITHUB_TOKEN or gh CLI credentials. This is a test-environment/credential limitation, not a product defect — GitHub's API correctly requires a token for higher rate limits and write actions like starring a repo. The agent's own --star code path was never actually exercised with a real token. Event 4: ```text env | grep -i -E 'gh_|github|token' | sed 's/=.*/=/'; which gh git node python3; gh --version 2>/dev/null ``` Event 38: ```text **No `GITHUB_TOKEN` in this environment**, so I could only exercise read-only endpoints. The `--star` write path is written and guarded but untested against a real token ``` - **gh CLI not installed in sandbox**: The `gh` CLI binary was absent from the environment (exit code 127), forcing the agent to fall back to the Octokit SDK path. This is a test-environment limitation rather than a product issue, and the agent noted the SDK was an acceptable alternative per the task's instructions. Event 6: ```text Command exited with code 127 ``` Event 38: ```text **`gh` CLI is not installed**, so I used the SDK path (which was your stated preference anyway). ``` #### Suggested Changes None identified in this transcript. ### Task given to each agent Help me build a simple example using GitHub. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: B · 84/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a text/markdown request; no Markdown representation offered. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links docs APIs, MCP server, CLI, REST/GraphQL and Copilot guides with usage guidance. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt 'How to use' section explains API-based retrieval and browsing docs pages. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt mentions REST/GraphQL APIs, GitHub MCP server, and agent skills docs. ``` - **Pass** — A compact guide representation exists ```text llms.txt is a compact Markdown guide with organized sections and API/MCP/CLI links. ``` - **Pass** — A focused guide is directly retrievable ```text llms.txt and CLI quickstart Markdown are directly retrievable as text/markdown. ``` - **Pass** — Equivalent instructions fit a token budget ```text llms.txt is 6220 tokens, under the 8000-token budget for a useful guide. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be judged. ``` - **Pass** — The compact guide is independently actionable ```text GitHub CLI quickstart gives concrete steps: install, `gh auth login`, then commands like `gh repo clone`. ``` - **Pass** — Install and next-step links resolve ```text Sampled install and next-step links (CLI install, docs, manual) fetched successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text llms.txt links GitHub CLI quickstart; fetched quickstart gives concrete first steps. ``` - **Pass** — Installation commands are extractable ```text Install pages give extractable commands: brew install gh, apt, dnf, winget. ``` - **Pass** — Code examples are available without interaction ```text Quickstart shows inline shell examples like gh auth login and gh repo clone. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Quickstart lists prerequisites and gh auth login; MCP guide requires a PAT. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders Free $0, Team $4, Enterprise $21 per user/month without interaction. ``` - **Pass** — Prices are stated, not gated ```text Prices stated directly: Free $0, Team $4, Enterprise $21 per user/month. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: per user/month, plus CI/CD minutes, storage GB, compute $0.18/hr. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Final output gives seat pricing (Free/Pro ~$4/Team ~$4/Enterprise ~$21) plus metered Actions/Codespaces rates, explicitly caveated as approximate and checked against docs/pricing page (seq 29 curl to github.com/pricing). Kimi K3: Final output states API access is free, breaks down rate-limit tiers (60/5,000/15,000 req-hr) and paid plan costs (~$4 and ~$21/user/mo), with explicit assumption that none of this example's usage triggers metered Actions/Packages/Codespaces costs. Qwen 3.8 Max: Final output breaks down API rate-limit tiers (unauthenticated vs token) plus per-user/month plan pricing (Free/Pro ~$4/Team/Enterprise ~$21, Copilot tiers), explicitly flagged as approximate/verify-at-github.com/pricing. This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text llms.txt links REST and GraphQL API docs; docs.github.com/api endpoints return structured JSON. ``` - **Pass** — An MCP server is documented and well-formed ```text GitHub MCP Server documented with remote/local install, toolsets, tools, and auth. ``` - **Pass** — A CLI install path is documented ```text GitHub CLI quickstart documents install, gh auth login, and core commands. ``` - **Skipped** — SDK packages resolve on their registries ```text No registry lookup result for a GitHub SDK/CLI package was supplied. ``` - **Pass** — Agent skills are published ```text Agent skills documented: SKILL.md format, directories, and gh skill install/update/publish. ``` [Full report data](https://www.ax-check.com/github.com/report.json)