# AX Check: graphql-ruby.org Checked 2026-09-19. Getting Started guide is clear, but no LLM-friendly docs index Quickstart with install steps and code examples works well (9 of 23 checks pass), but /llms.txt returns 404 and homepage has no Markdown version. ## End-to-end onboarding not demonstrated ## Coding sessions All three independent agent sessions completed successfully, confirming the core gem is free under MIT license. Each noted the separate GraphQL::Pro add-on has paid pricing, though quoted figures varied between sessions. ### DeepSeek V4 Pro [View public run](https://agents.withgauge.com/p/runs/d3cc27eb-07da-4cd4-b038-f08f151c1847) · [Read transcript](https://www.ax-check.com/graphql-ruby.org/sessions/deepseek.json) Final output states core gem is free/MIT and GraphQL::Pro is $1,100/year 'per graphql.pro', explicitly noting it's a self-hosted library add-on rather than a hosted service — pricing tied to a stated source and plan context. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: Agent only fetched the public graphql.pro marketing page via curl to check pricing. No credentials were obtained, no account was created, and no authenticated API/SDK call was made. Agent explicitly concluded there is no hosted product to act on and stopped without attempting any onboarding flow. Event 5: ```text https://graphql.pro/ ``` Event 23: ```text GraphQL Ruby has no hosted API and no SDK endpoint I can call from here. The entire "developer workflow" is: add the gem to a Ruby/Rails app, define your schema in Ruby, serve it from *your own* Rack/Rails server. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No hosted API/SDK exists for GraphQL Ruby or GraphQL::Pro**: GraphQL Ruby (and its paid add-on GraphQL::Pro) is a library installed into a user's own Ruby/Rails application, not a hosted service with an API or SDK. Per the task instructions to stay light and avoid local service stacks, the agent correctly identified it could not exercise a real developer workflow without spinning up a local Rack/Rails server (and Redis for Pro subscriptions), which it was told not to do. This is a product-nature limitation (no hosted offering) rather than a credential or agent-error issue, and the agent stopped per instructions instead of fabricating a workaround. Event 23: ```text There is nothing hosted for me to "use through its API." ``` Event 23: ```text The quickstart for graphql-ruby requires a local service stack and running a server, which you told me not to do. ``` #### Suggested Changes - **Add explicit hosted-vs-library distinction to graphql.pro homepage**: On graphql.pro's landing page, add a clear line near the pricing/buy button clarifying that GraphQL::Pro is a library to install into a self-hosted Ruby/Rails app, not a hosted API service. This would let evaluators and first-time developers immediately understand the deployment model instead of having to infer it from feature descriptions. Verify by re-checking the page text fetched at https://graphql.pro/ for an explicit 'self-hosted' or 'installed into your app' statement near the top. Event 21: ```text GraphQL::Pro is a toolkit for the GraphQL-Ruby gem . Take your schema to the next level! Buy @ $1100/year ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/3ff64334-4557-461b-aaba-b8c2ebc9c27a) · [Read transcript](https://www.ax-check.com/graphql-ruby.org/sessions/kimi.json) Final output states the core gem costs $0 (MIT license, no account/API key) and explicitly scopes that to 'the core library used here,' distinguishing it from the paid GraphQL::Pro add-on rather than quoting an unfounded number. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: GraphQL Ruby is a self-hosted open-source library, not a hosted product with credentials or an authenticated API. The agent never attempted to obtain credentials or perform an authenticated operation against a hosted service; it installed the gem locally and ran an in-process query against static in-memory data. No product account, API key, or hosted database was involved. Event 29: ```text { "data": { "products": [ { "id": "1", "name": "Coffee Mug", ``` #### Hallucinated URLs None identified in this transcript. #### Blockers None identified in this transcript. #### Suggested Changes - **Clarify that graphql-ruby has no hosted product or developer workflow**: The seed prompt asked how pricing works and whether the product would be easy to manage via hosted SDK/API, but graphql-ruby is a plain open-source gem with no account, API key, or hosted service (the only paid tier is the optional GraphQL::Pro add-on). Product-facing docs or landing pages could state upfront that there is no hosted offering, so evaluators/users expecting an SDK-and-API workflow understand it is a local library instead. Verify by checking that the graphql-ruby homepage or getting-started guide explicitly notes it runs entirely in-process with no server or account required. Event 31: ```text GraphQL Ruby (the `graphql` gem) is free and open source under the MIT license. There's no hosted product, no account, and no API key. ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/b42957ae-c83c-4f04-836a-75f4cb71d828) · [Read transcript](https://www.ax-check.com/graphql-ruby.org/sessions/qwen.json) Final output states core gem is free/MIT, GraphQL-Pro is '~$150+/developer, subscription' and Enterprise is custom pricing, tying each figure to a plan-tier assumption rather than a bare number. #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: Agent never attempted to obtain or use product credentials. It determined immediately that GraphQL Ruby is a local, open-source library with no hosted API/SDK, and stopped per the seed prompt's instruction to halt if there is no actionable hosted developer workflow. No credential acquisition or authenticated operation occurred. Event 12: ```text There is no hosted service, no API endpoint, no SDK-to-cloud, and no account or API key involved. ``` Event 12: ```text Plainly: GraphQL Ruby has no hosted developer workflow I can act on, so I'm stopping here. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Ruby runtime installed in sandbox**: The test environment lacked a Ruby toolchain (ruby, gem, bundle all missing), which would have been needed to run any local GraphQL Ruby example. This is an environment limitation rather than a product defect, and became moot once the agent determined GraphQL Ruby is local-only and out of scope per the task's hosted-product constraint. Event 5: ```text /bin/bash: line 1: ruby: command not found /bin/bash: line 1: gem: command not found Command exited with code 127 ``` - **GraphQL Ruby has no hosted API/SDK to exercise**: Product behavior: GraphQL Ruby is a self-hosted, in-process library with no cloud endpoint, account, or API key. Given the session's explicit instruction to stay light and use only a hosted SDK/API, the agent correctly identified this as an unactionable workflow and stopped rather than standing up a local stack. Event 12: ```text It's not a hosted product — it's an open-source Ruby *library* (`graphql` gem) that runs in-process inside your own Ruby/Rails application. ``` Event 12: ```text Provisioning a Ruby runtime + gems via apt would be exactly the kind of heavyweight local setup you asked me to avoid, and even then the result would just be local code execution, not interaction with a product I can manage. ``` #### Suggested Changes None identified in this transcript. ### Task given to each agent Help me build a simple example using GraphQL Ruby. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: C · 55/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a Markdown Accept header; no Markdown representation offered. ``` - **Failed** — llms.txt provides an actionable documentation index ```text /llms.txt returns HTTP 404, so no documentation index exists. ``` - **Failed** — A compact guide representation exists ```text No site-published Markdown guide; /llms.txt returns 404 and homepage serves HTML only. ``` - **Skipped** — llms.txt provides navigation guidance ```text /llms.txt is 404, so no navigation guidance can be evaluated. ``` - **Skipped** — llms.txt mentions offered API, MCP, and skills ```text /llms.txt is 404, so no API/MCP/skills mentions can be evaluated. ``` - **Pass** — A focused guide is directly retrievable ```text Getting Started page is directly retrievable with install, type, schema, and query steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text Getting Started guide is 1365 tokens, well under the 8000-token budget. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be measured. ``` - **Pass** — The compact guide is independently actionable ```text Getting Started gives concrete steps: add gem, bundle install, generators, declare types, build schema, execute queries. ``` - **Pass** — Install and next-step links resolve ```text Install and next-step links resolve: getting_started, guides, and API doc pages fetched successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Getting Started page gives concrete steps: install gem, run generators, declare types, build schema, execute queries. ``` - **Pass** — Installation commands are extractable ```text Installation shows `gem "graphql"` in Gemfile and `bundle install`, plus rails g graphql:install. ``` - **Pass** — Code examples are available without interaction ```text Full Ruby code examples for types, schema, resolver and query execution appear inline without interaction. ``` - **Skipped** — Prerequisites and auth boundaries are explicit ```text Getting Started covers Rails generators and manual setup but states no explicit prerequisites or auth boundaries. ``` ### Pricing - **Skipped** — Pricing is readable without interaction ```text Not applicable: the relevant product is free. GraphQL-Ruby is a free open-source gem; no pricing page or paid plans were fetched. ``` - **Skipped** — Prices are stated, not gated ```text Not applicable: the relevant product is free. No pricing page fetched; the gem is free and Pro is a separate product. ``` - **Skipped** — Pricing units and limits are explicit ```text Not applicable: the relevant product is free. No pricing units or limits exist for this free open-source library. ``` - **Skipped** — Agents identify pricing and its assumptions ```text Not applicable: the relevant product is free. GraphQL-Ruby is a free open-source gem; no pricing page or paid plans were fetched. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Nav links to API reference at /api-doc/2.6.10/ on the getting-started page. ``` - **Skipped** — An MCP server is documented and well-formed ```text No MCP server documentation found in the fetched pages. ``` - **Pass** — A CLI install path is documented ```text Getting Started documents Rails generators: rails g graphql:install and graphql:object. ``` - **Skipped** — SDK packages resolve on their registries ```text No registry lookup result for the graphql gem was supplied. ``` - **Skipped** — Agent skills are published ```text No agent skills are documented in the fetched pages. ``` [Full report data](https://www.ax-check.com/graphql-ruby.org/report.json)