avatar > My API Key). exit=1 ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No JumpCloud account or API key available in sandbox**: The environment had no JumpCloud credentials pre-provisioned (env grep returned nothing), and the agent has no self-service way to create a JumpCloud org/account or mint an API key from within the sandbox. This is a test-environment/credential limitation, not a product defect — JumpCloud's API correctly requires a valid key (401 on bad key, redirect to login on unauthenticated browser-style request). Event 5: ```text (no output) ``` Event 11: ```text 302 Found. Redirecting to /login ``` Event 16: ```text {"error":"Unauthorized","message":"Unauthorized: api key user not found"} HTTP 401 ``` #### Suggested Changes None identified in this transcript. ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/cad68ebb-6f76-432a-bf12-e2d6164b1b0d) · [Read transcript](https://www.ax-check.com/docs.jumpcloud.com/sessions/qwen.json) README.md and final summary give per-user/month pricing tiers with explicit assumptions: annual vs monthly billing, à-la-carte vs bundled, 30-day trial, suspended-user billing, and a note to verify current terms. #### End-to-end onboarding - **Onboarding needs a login**: The agent never obtained real JumpCloud API credentials. It searched PyPI/npm for an SDK, confirmed the hosted REST API responds with clean JSON (401 for unauthenticated and invalid-key requests), and built a Python client plus a runnable example, but no JUMPCLOUD_API_KEY was available in the environment and the agent never acquired one itself. It tested only a probe (unauthenticated) path and a deliberately invalid-key path, both of which returned 401 Unauthorized. No authenticated operation (e.g. listing real users, creating a user) against the hosted product was ever demonstrated. Event 6: ```text ---ENV (filtered)--- PI_GATEWAY_API_KEY= ---done--- ``` Event 90: ```text HTTP 401 -> {'message': 'Unauthorized: user not authenticated'} ``` Event 95: ```text API error: [HTTP 401] Unauthorized: api key user not found. Invalid or missing API key. Set a valid admin x-api-key (JUMPCLOUD_API_KEY). ``` Event 102: ```text no `JUMPCLOUD_API_KEY` is set, so I can only probe (confirm reachability + the 401), not actually manage your directory. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No admin API key available to authenticate against JumpCloud**: Every useful JumpCloud v2 endpoint requires an x-api-key from an admin account. The sandbox environment had no JUMPCLOUD_API_KEY set and the agent has no path to self-serve one (would require signing up for a trial/paid tenant and logging into the console). This is a normal credential requirement, not a product defect — the API responded correctly and predictably (clean JSON 401) in both the no-key and invalid-key cases. Event 24: ```text pypi jumpcloud: 200 ``` Event 65: ```text HTTP 401 | content-type=application/json; charset=utf-8 --- body (first 300 chars) --- {"message":"Unauthorized: user not authenticated"} ``` Event 102: ```text no `JUMPCLOUD_API_KEY` is set, so I can only probe (confirm reachability + the 401), not actually manage your directory. ``` - **No official JumpCloud SDK cleanly published on PyPI or npm**: The agent searched npm and PyPI for an official JumpCloud SDK and found only unrelated or third-party packages (a home-assistant skill under the name jcapi, a third-party CLI named jumpcloud). This is a product packaging/discoverability gap rather than an agent error, since the agent correctly identified the mismatches before falling back to the documented REST API. Event 50: ```text name: jcapi version: 0.1.5 summary: home-assistant skill api from jiachang. author: ewg urls: {'Homepage': 'https://github.com/iessen/jcapi.git'} ``` Event 32: ```text name: jumpcloud version: 2.0.1 summary: jumpcloud command for jumpcloud.com author: Karl Rink ``` Event 53: ```text So there's **no cleanly‑published official JumpCloud SDK** on PyPI/npm — the official ones live on GitHub (`TheJumpCloud/jcapi-*`) as generated API clients. ``` #### Suggested Changes - **Publish an official JumpCloud SDK under an unambiguous name on PyPI and npm**: Searching npm and PyPI for 'jumpcloud' surfaced only third-party/unrelated packages (a home-assistant skill under 'jcapi', an unofficial CLI under 'jumpcloud'). Publish the GitHub TheJumpCloud/jcapi-* generated clients to PyPI/npm under a clearly namespaced package (e.g. an official org scope) so `pip install` or `npm install` surfaces the real SDK first. Verify by searching the registries for 'jumpcloud' and confirming the top result is the official client. Event 50: ```text name: jcapi version: 0.1.5 summary: home-assistant skill api from jiachang. ``` Event 32: ```text name: jumpcloud version: 2.0.1 summary: jumpcloud command for jumpcloud.com author: Karl Rink ``` - **Clarify current free-tier / trial terms on the pricing page**: The pricing page fetch found a 30-day trial mention but no explicit free-tier language, leaving the agent to flag outdated assumptions about a historical ~10 free users offer. Add an explicit statement on jumpcloud.com/pricing about whether any permanent free tier exists today, and confirm by re-fetching the page and searching for 'free' near the trial section. Event 59: ```text [30-day] ...o enterprise. Pick the JumpCloud products you need. Start a 30-day trial. Find the package that fits your needs. Contact our team. See À La Carte... ``` Event 82: ```text No permanent free tier is advertised on the current pricing page (JumpCloud historically offered free usage for up to ~10 users — confirm current terms before relying on it). ``` ### Task given to each agent Help me build a simple example using JumpCloud. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: D · 29/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html even when Markdown was requested; no Markdown representation offered. ``` - **Failed** — llms.txt provides an actionable documentation index ```text https://docs.jumpcloud.com/llms.txt returns HTTP 404, so no documentation index exists. ``` - **Skipped** — llms.txt provides navigation guidance ```text llms.txt is 404, so its navigation guidance cannot be evaluated. ``` - **Skipped** — llms.txt mentions offered API, MCP, and skills ```text llms.txt is 404, so its mention of API, MCP, or skills cannot be evaluated. ``` - **Skipped** — A compact guide representation exists ```text No compact agent-specific guide representation was fetched; only API index pages and a YAML mapping. ``` - **Skipped** — A focused guide is directly retrievable ```text No focused quickstart or guide page was fetched; only API version index pages. ``` - **Skipped** — Equivalent instructions fit a token budget ```text No compact guide was fetched, so token budget cannot be measured. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown is unsupported, so link preservation across formats is unassessed. ``` - **Skipped** — The compact guide is independently actionable ```text No compact agent guide fetched; only API index pages and a YAML mapping were retrieved. ``` - **Pass** — Install and next-step links resolve ```text Fetched API index pages and the operation-id-mapping.yaml all returned HTTP 200. ``` ### Onboarding - **Skipped** — Docs lead to a relevant quickstart ```text Only a 404 llms.txt was fetched; no docs or quickstart pages were retrieved. ``` - **Skipped** — Installation commands are extractable ```text No installation or setup pages were fetched to extract commands from. ``` - **Skipped** — Code examples are available without interaction ```text No documentation pages with code examples were fetched. ``` - **Skipped** — Prerequisites and auth boundaries are explicit ```text No auth or prerequisites documentation was fetched for assessment. ``` ### Pricing - **Skipped** — Pricing is readable without interaction ```text No pricing page was fetched; only API documentation homepage evidence is available. ``` - **Skipped** — Prices are stated, not gated ```text No pricing page was fetched; only API documentation homepage evidence is available. ``` - **Skipped** — Pricing units and limits are explicit ```text No pricing page was fetched; only API documentation homepage evidence is available. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Pricing table cites per-user/month figures scraped live from jumpcloud.com/pricing (seq 26/42) and final output notes assumptions like annual vs monthly, add-on costs, free trial, and quote-based enterprise tiers. Kimi K3: Final output gives concrete tiered pricing (free up to 10 users/devices, ~$11/user/mo core, ~$19-24/user/mo full platform) and flags it as approximate, billed-annually, bundle/volume-based assumptions. Qwen 3.8 Max: README.md and final summary give per-user/month pricing tiers with explicit assumptions: annual vs monthly billing, à-la-carte vs bundled, 30-day trial, suspended-user billing, and a note to verify current terms. This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Docs describe OpenAPI spec bundles and direct spec download URLs at docs.jumpcloud.com/new. ``` - **Pass** — An MCP server is documented and well-formed ```text MCP server documented: endpoints, Streamable HTTP transport, API key/OAuth auth, supported clients and tools. ``` - **Skipped** — A CLI install path is documented ```text No CLI install path for JumpCloud's own tooling appears in the fetched documents. ``` - **Skipped** — SDK packages resolve on their registries ```text No registry lookup for a JumpCloud SDK or packaged CLI package was supplied. ``` - **Skipped** — Agent skills are published ```text No agent skills publication is offered or evidenced in the fetched documents. ``` [Full report data](https://www.ax-check.com/docs.jumpcloud.com/report.json)