# AX Check: airops.com Checked 2026-09-21. Docs and quickstart work, but plan prices stay hidden Solo and Pro tiers say "Start for free" instead of dollar prices; only usage limits and overage rate ($0.025/task) are stated. ## Onboarding needs a login ## Coding sessions All three independent sessions completed. DeepSeek V4 Pro and Kimi K3 each pulled Solo/Pro/Enterprise task-based pricing straight from the live pricing page, noting trial credits and overage rates as assumptions; Qwen 3.8 Max completed without a recorded pricing writeup. ### DeepSeek V4 Pro [View public run](https://agents.withgauge.com/p/runs/3e5b73d1-5517-45f1-9c8d-66ee895a200d) · [Read transcript](https://www.ax-check.com/airops.com/sessions/deepseek.json) Final output gives Solo/Pro/Enterprise task-based pricing with explicit assumptions (task-based usage, trial credits, overage rate, 'figures... can change, treat as currently documented'). #### End-to-end onboarding - **Onboarding needs a login**: The agent explored AirOps docs extensively and built a working Python script targeting the hosted REST API, but never obtained real API credentials. It only had access to a Fireworks gateway key (unrelated to AirOps) and explicitly stated it lacked an AirOps API key or a published app UUID. It tested unauthenticated and fake-key requests against the live API, both correctly returning 401, which proves the API is reachable but does not demonstrate an authenticated operation succeeding. Event 7: ```text PI_GATEWAY_API_KEY= GIT_CONFIG_KEY_0= ``` Event 90: ```text API error 401: {"error":"HTTP Token: Access denied"} ``` Event 111: ```text I can't complete a live round-trip because I don't have your AirOps API key or a published app UUID (nothing in this environment). ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No AirOps account credentials available in the sandbox**: The session environment only exposed a Fireworks gateway key, not an AirOps API key or workspace login. Since AirOps requires signing in at app.airops.com to get a Bearer API key and to create/publish an app, the agent could not complete a live authenticated call. This is a normal login/credential requirement, not a product defect. Event 7: ```text PI_GATEWAY_API_KEY= GIT_CONFIG_KEY_0= ``` Event 111: ```text I can't complete a live round-trip because I don't have your AirOps API key or a published app UUID (nothing in this environment). ``` - **App creation/publishing is UI-only, not API-driven**: The REST API can list and execute already-published apps, but the docs show no endpoint to create or build a Workflow/Agent app programmatically — that must be done in the app.airops.com UI first. This limited how far a pure API/SDK exercise could go without a human building an app in the UI beforehand. Event 111: ```text Auth is a simple Bearer token, endpoints are straightforward (`execute`, `async_execute`, `webhook_async_execute`, agent `chat`, executions lookup), and the docs publish machine-readable `.md`/`llms.txt` versions. The one friction point is build-side: you create/publish the app itself in the **UI**, not via the API ``` #### Suggested Changes - **Add a note on the API Authentication page that a published app must exist first**: On https://docs.airops.com/api-reference/using-the-api/authentication (and/or the Executions/Apps API pages), state upfront that execute/async_execute endpoints require an already-published app created in the app.airops.com UI, since the API itself offers no app-creation endpoint. Verify by having a first-time developer read only the API reference and confirm they know to go build an app in the UI before calling execute. Event 33: ```text Preconditions: the app must be published . ``` Event 111: ```text the REST API lists/executes published apps but doesn't build workflows programmatically ``` - **Clarify the pricing FAQ task-count discrepancy between plan cards and FAQ example**: The pricing page lists Solo at 35,000 tasks and Pro at 100,000 tasks in the plan cards, but a separate on-page element shows 1,000 tasks/month and 10,000 tasks/month. Reconcile these figures on https://www.airops.com/pricing so a developer estimating cost isn't confused about actual plan limits. Event 51: ```text 35,000 Tasks for Content Production 1 Brand Kit, 3 Knowledge Bases ``` Event 78: ```text 1,000 tasks/month 10,000 tasks/month ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/2f151990-5faf-4caa-8c90-398b43f1447a) · [Read transcript](https://www.ax-check.com/airops.com/sessions/kimi.json) Final output gives task-based pricing (Solo 35k tasks free, $0.025/task overage; Pro 100k tasks; Enterprise custom) explicitly tied to plan tier and trial/reset assumptions scraped from the live pricing page (seq 22-25). #### End-to-end onboarding - **Onboarding needs a login**: The agent researched the AirOps docs and API reference thoroughly, wrote a working Python script that calls the hosted execute/poll endpoints, and confirmed the API is live by hitting it unauthenticated (got a 401 as expected). However, it never obtained a real API key or workspace: signup requires an email/SSO the agent doesn't have, and it explicitly stopped rather than proceeding. No authenticated call to the actual product was ever made. Event 42: ```text 404 https://app.airops.com/signup ``` Event 54: ```text unauthenticated execute -> HTTP 401 ``` Event 55: ```text I can't run a live end-to-end call. AirOps has no unauthenticated tier — the API requires a Bearer key from Workspace Settings, and signup at app.airops.com needs an email/SSO I don't have. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service signup path found for API key**: The agent tried the direct signup URL and got a 404, and could not find any email/SSO signup flow it could complete without human input. This is a missing-credentials blocker rather than a product defect — normal login/signup is expected to require a human, and the agent correctly identified this rather than fabricating a workaround. Event 42: ```text 404 https://app.airops.com/signup ``` Event 55: ```text signup at app.airops.com needs an email/SSO I don't have ``` - **Workflows can only be authored in the GUI, not via API**: Per AirOps' own docs, workflows are built and published only in the web app; the API can execute and poll already-published workflows but cannot create or version them. This is a product design characteristic (not agent error) that limits how much of the developer workflow can be automated headlessly. Event 41: ```text After publishing your workflow, you can call it through our API endpoints using your required input parameters to receive a response. ``` Event 55: ```text there's no way to author a workflow programmatically — workflows are built in the GUI, published, and only executed via API ``` #### Suggested Changes - **Add a working direct signup link to the docs and marketing site**: The agent's attempt to reach https://app.airops.com/signup returned a 404. Point the 'Start for free' pricing CTA (or the docs Getting Started page) at a live signup URL so a new developer landing from docs.airops.com doesn't hit a dead end. Verify by curling the linked signup URL and confirming a 200 response with an actual signup form. Event 42: ```text 404 https://app.airops.com/signup ``` - **Fix the broken API Reference link inside the Run via API doc page**: The 'Run via API' page links to the API Reference using a broken placeholder URL scheme instead of a real docs link, which would send a developer following the doc to a dead reference. Replace 'broken://spaces/rT8WZaGa5JcV0M5osTWg' with the actual https://docs.airops.com/api-reference path and confirm it resolves. Event 41: ```text check out our [API Reference](broken://spaces/rT8WZaGa5JcV0M5osTWg) for detailed technical specifications ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/46790d03-8336-4928-ba2c-1e9f2ec49e19) · [Read transcript](https://www.ax-check.com/airops.com/sessions/qwen.json) #### End-to-end onboarding - **Onboarding needs a login**: The agent never obtained real AirOps credentials on its own. It searched the environment for keys/tokens, found none, and confirmed via probing that the live API rejects placeholder credentials with 401. It built a complete example (client, preflight, run scripts) but exercised it only with dummy values, producing expected 401s, not a successful authenticated operation. The agent states signup/API-key retrieval requires a human to create an account and copy the key from Settings, and asks the user to supply real credentials to finish the loop. Event 5: ```text PI_GATEWAY_API_KEY= GIT_CONFIG_KEY_0= ``` Event 113: ```text Checking app 1 ... FAILED (HTTP 401): Not Authorized ``` Event 145: ```text Getting a key requires a human: create an account at `app.airops.com`, then Settings → Workspace → API KEY. I can't sign up on your behalf. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service way to obtain AirOps workspace credentials**: The session has no AirOps API key, workspace ID, or app ID pre-provisioned, and there is no public signup/API flow the agent can drive itself — account creation happens through the app.airops.com web UI. This is a normal login/credential requirement, not a product defect, and it fully blocked any authenticated execution of a real app. Event 145: ```text No AirOps credentials exist in this environment (no workspace id, API key, or token). ``` Event 145: ```text Getting a key requires a human: create an account at `app.airops.com`, then Settings → Workspace → API KEY. I can't sign up on your behalf. ``` - **No public/demo AirOps app available to test without auth**: The agent checked whether the SDK's unauthenticated 'public app' execution path could demonstrate a real call without a workspace key, but every app_base ID it probed (1, 2, 100) returned 404, meaning there was no discoverable public app to run against. Event 44: ```text --- app_base 1 --- {"error":"Not Found"} HTTP 404 ``` Event 44: ```text === async_execute no auth === {"error":"Not Authorized"} HTTP 401 ``` - **JS SDK is browser-targeted and crashes on plain Node import**: The official @airops/airops-js package bundles pusher-js's browser build, which references window/document at import time. Running it directly in Node.js throws a ReferenceError, forcing the agent to write custom DOM shims before the SDK would even load — a product packaging issue rather than agent error, since the npm package is advertised as a general JS SDK. Event 48: ```text ReferenceError: window is not defined at /tmp/probe/node_modules/@airops/airops-js/dist/index.cjs.js:114:5 ``` Event 60: ```text SDK loaded in Node OK. apps methods: get, execute, chatStream, getResults, cancelExecution, pollExecutionResult, initializePusher ``` #### Suggested Changes - **Fix Node.js compatibility of @airops/airops-js so it imports without DOM shims**: Running require('@airops/airops-js') in a bare Node.js script throws ReferenceError: window is not defined because the bundled pusher-js browser build touches window/document at import time (dist/index.cjs.js). Ship a Node-safe build, or lazy-load the Pusher/browser transport only when streaming is requested, so a fresh npm install and require succeeds without custom shims. Verify by importing the package on a clean Node 22 environment. Event 48: ```text ReferenceError: window is not defined at /tmp/probe/node_modules/@airops/airops-js/dist/index.cjs.js:114:5 ``` - **Add a public/sandbox demo app for unauthenticated SDK testing**: The airops-js README states public apps can be executed without authorization, but probing app_base IDs 1, 2, and 100 at the sdk_api/airops_app_bases endpoint all returned 404. Publish at least one real public demo app ID in the docs so new developers can exercise apps.execute/apps.get end-to-end before creating a workspace. Verify by confirming the documented public app ID returns 200 from an unauthenticated request. Event 44: ```text --- app_base 1 --- {"error":"Not Found"} HTTP 404 ``` - **Expose task/usage quota in the execution response or a lightweight API endpoint**: Pricing is billed per task with overage charged at $0.025/task, but the SDK's execute/getResults responses carry no quota or remaining-tasks field. Add a machine-readable usage/quota field to the execution status response, or a small usage endpoint, so scripts can budget task consumption programmatically instead of only checking Settings > Billing in the UI. Verify by confirming a poll of an execution or new usage endpoint returns current task counts. Event 23: ```text You can easily track your task usage through the AirOps platform. From app.airops.com follow the navigation path Settings>Billing and you will see the tasks you’ve used so far in the monthly billing period ``` ### Task given to each agent Help me build a simple example using AirOps. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: C · 60/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a Markdown Accept header; no Markdown representation offered. ``` - **Failed** — llms.txt provides an actionable documentation index ```text https://airops.com/llms.txt returned the HTML homepage, not a documentation index. ``` - **Skipped** — llms.txt provides navigation guidance ```text No llms.txt body was served, so navigation guidance cannot be assessed. ``` - **Skipped** — llms.txt mentions offered API, MCP, and skills ```text No llms.txt body was served, so API/MCP/skills mentions cannot be assessed. ``` - **Pass** — A compact guide representation exists ```text Docs pages serve Markdown via .md suffix, e.g. getting-started.md and core-concepts.md. ``` - **Pass** — A focused guide is directly retrievable ```text Getting Started guide is directly retrievable as Markdown with concrete Playbook and Workflow steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text Getting Started Markdown is 641 tokens, well under the 8000-token budget. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be measured. ``` - **Pass** — The compact guide is independently actionable ```text Getting Started gives concrete steps: build a first Playbook or Workflow with linked guides. ``` - **Pass** — Install and next-step links resolve ```text Fetched install/next-step docs (getting-started, core-concepts, MCP, API) all returned 200. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Getting Started links to Building your first Playbook and first Workflow quickstarts with concrete steps. ``` - **Skipped** — Installation commands are extractable ```text AirOps is a hosted platform; no installation commands are offered in the fetched docs. ``` - **Pass** — Code examples are available without interaction ```text Docs show inline code examples, e.g. the GET ask query snippet, without interaction. ``` - **Skipped** — Prerequisites and auth boundaries are explicit ```text Fetched docs describe platform concepts but no API auth or key prerequisites. ``` ### Pricing - **Failed** — Prices are stated, not gated ```text Solo and Pro show 'Start for free' with no dollar prices; only Enterprise is contact-sales. ``` - **Pass** — Pricing is readable without interaction ```text Pricing page renders Solo, Pro, Enterprise tiers with feature lists directly in HTML, no interaction needed. ``` - **Pass** — Pricing units and limits are explicit ```text Task limits (35,000/100,000), seats, brand kits and $0.025 per extra task are stated. ``` - **Pass** — Agents identify pricing and its assumptions ```text 2 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4 Pro: Final output gives Solo/Pro/Enterprise task-based pricing with explicit assumptions (task-based usage, trial credits, overage rate, 'figures... can change, treat as currently documented'). Kimi K3: Final output gives task-based pricing (Solo 35k tasks free, $0.025/task overage; Pro 100k tasks; Enterprise custom) explicitly tied to plan tier and trial/reset assumptions scraped from the live pricing page (seq 22-25). This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Insights API reference with endpoints, auth, and schemas is reachable at docs.airops.com/api-reference. ``` - **Pass** — An MCP server is documented and well-formed ```text MCP server documented: URL https://app.airops.com/mcp, OAuth, per-client setup, tools reference. ``` - **Skipped** — A CLI install path is documented ```text No CLI install path documented; MCP client setup uses npx mcp-remote, not a product CLI. ``` - **Skipped** — SDK packages resolve on their registries ```text No SDK or packaged CLI registry lookup supplied; only API and MCP surfaces documented. ``` - **Skipped** — Agent skills are published ```text No agent skills published; docs cover API and MCP only, no skills surface offered. ``` [Full report data](https://www.ax-check.com/airops.com/report.json)