# AX Check: valohai.com Checked 2026-11-05. Valohai has a quickstart guide, but hides its actual prices A migration guide with real commands is easy to fetch, but the pricing page never states a number, only a per-user license sold by custom quote. ## Onboarding needs a login ## Coding sessions All three independent sessions read the pricing page and reported a per-user license with compute billed separately, but none could find an actual price since the page only offers custom quotes with no public figure. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/a7e3af17-0ce6-4513-805a-941cd64b65df) · [Read transcript](https://www.ax-check.com/valohai.com/sessions/deepseek.json) Gave no made-up number. Based on the pricing page, it said pricing is a fixed per-user monthly fee by custom quote with no public price list, platform usage is unlimited, and compute is billed separately. (seq 38, seq 124, seq 126) #### End-to-end onboarding - **Onboarding needs a login**: No Valohai account or token existed in the sandbox. The agent confirmed the hosted REST API rejects unauthenticated requests (403 not_authenticated) and that vh login is interactive/browser-based, so it could not obtain credentials itself. It authored and locally validated a Valohai example (vh lint, local python train.py dry-run) but never executed an authenticated operation against the real hosted Valohai API or created any hosted resource. Event 39: ```text --- valohai config files --- --- any tokens in env --- none ``` Event 40: ```text projects: 403 {"message":"Authentication credentials were not provided.","code":"not_authenticated"} ``` Event 126: ```text This environment has no Valohai credentials, and protected endpoints return `403 not_authenticated`. `vh login` is interactive/browser-based, so I can't complete it. I can't submit or observe a real hosted run without an account/token. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No credentials available to authenticate with Valohai's hosted platform**: The sandbox had no Valohai account, token, or CLI config. vh login requires an interactive browser flow or a personal access token, neither of which was obtainable without human involvement, so the agent could not submit or observe a real hosted execution. This is a session/credential limitation, not a product defect — login is a normal requirement. Event 39: ```text --- any tokens in env --- none ``` Event 40: ```text {"message":"Authentication credentials were not provided.","code":"not_authenticated"} ``` Event 126: ```text `vh login` is interactive/browser-based, so I can't complete it. I can't submit or observe a real hosted run without an account/token. ``` - **Quickstart example code references a nonexistent SDK function**: The official quickstart's train.py sample calls valohai.log_metric, but the installed valohai-utils 0.8.0 package has no such attribute (log_metric is None). The agent discovered this via local introspection and used valohai.logger() instead — a doc/code mismatch in the product's own quickstart, not an agent error. Event 57: ```text == log_metric == None sig err None is not a callable object ``` Event 68: ```text Note: the docs' `valohai.log_metric` doesn't exist in the installed `valohai-utils` 0.8.0 — the real API is `valohai.logger()`. I'll use the actual API. ``` - **Parameters have no value outside the hosted platform, causing a crash on first local run**: valohai.parameters('learning_rate').value returns None when run outside Valohai (no default supplied), causing a TypeError on the first local dry-run of the quickstart-style script. The agent recovered by passing explicit defaults to the Parameter constructor, but a first-time user following the quickstart literally would hit this crash immediately. Event 82: ```text TypeError: float() argument must be a string or a real number, not 'NoneType' ``` Event 88: ```text no default -> None with default -> 0.1 ``` #### Suggested Changes - **Fix the quickstart's train.py to use a real valohai-utils API**: On the Hello World quickstart page (docs.valohai.com/getting-started/quickstart), replace valohai.log_metric("accuracy", accuracy, step=epoch) with the actual API, valohai.logger() / logger.log(...), matching what ships in valohai-utils 0.8.0. Verify by pip installing valohai-utils and confirming the documented call exists via python -c "import valohai; valohai.log_metric" without raising AttributeError. Event 57: ```text == log_metric == None sig err None is not a callable object ``` Event 30: ```text valohai.log_metric("accuracy", accuracy, step=epoch) ``` - **Add a default value or explicit guidance for local parameter usage in the quickstart script**: In the quickstart's train.py sample, valohai.parameters("learning_rate").value has no default, so running the script outside Valohai raises a TypeError. Update the sample to pass a default (e.g. valohai.parameters("learning_rate", 0.001).value) or add a note that the script only works when run via `vh execution run`. Verify by running `python train.py` locally without a Valohai project linked and confirming it no longer crashes. Event 82: ```text TypeError: float() argument must be a string or a real number, not 'NoneType' ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/6828d43e-28f3-41aa-b5ac-9759ef749ef8) · [Read transcript](https://www.ax-check.com/valohai.com/sessions/kimi.json) Gave pricing grounded in the fetched page: per-user license, custom quote with no public figure, free trial, compute paid separately; did not invent a number (seq 24, seq 55) #### End-to-end onboarding - **Onboarding needs a login**: Agent never obtained real Valohai credentials. It confirmed the CLI (vh) was installed but had no token, attempted `vh login` interactively which aborted immediately because account creation requires the web UI, and the demo app.valohai.com API call returned 403. All subsequent work (train.py, valohai.yaml) was validated only with a local Python run, not against the hosted product. Event 50: ```text Login hostname? (You can just also accept the default https://app.valohai.com/ by leaving this empty.) If you don't yet have an account, please create one at https://app.valohai.com/ first. https://app.valohai.com/ - Username: Aborted! signup page HTTP 200 ``` Event 55: ```text I can't run the example on the hosted product from this sandbox. Authentication requires an account created through the web UI (with email verification), and there are no Valohai credentials in this environment — there's no API-only signup path. ``` Event 44: ```text 😄 Success! /sandbox/repo/valohai-example/valohai.yaml: No errors epoch= 1 mse=12.7559 w=0.769 b=0.186 ... saved model -> ./model.json ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service way to obtain Valohai API credentials**: Valohai requires account creation through the web UI before any API/CLI token can be issued. The agent tried `vh login` and it aborted waiting for a username, since no credentials existed in the sandbox and there was no programmatic signup path available. This is a normal authentication requirement (not a code defect), but it fully blocked any authenticated use of the hosted product. Event 50: ```text If you don't yet have an account, please create one at https://app.valohai.com/ first. https://app.valohai.com/ - Username: Aborted! ``` Event 55: ```text Authentication requires an account created through the web UI (with email verification), and there are no Valohai credentials in this environment — there's no API-only signup path. ``` - **Direct hosted API call rejected without auth**: An unauthenticated request to the hosted projects endpoint returned a 403, confirming the API requires credentials that were unavailable in-session. This is expected product behavior for a secured API, not an agent mistake. Event 7: ```text 403 Python 3.12.3 ``` #### Suggested Changes - **Publish a public per-seat price figure on the pricing page**: The pricing page (valohai.com/pricing) states pricing is per-user license but provides no actual dollar figures, pushing users to 'Get a custom quote.' Add at least an indicative starting price or price range on this page so prospective self-serve developers can gauge cost without contacting sales; verify by checking the scraped page text no longer requires a quote request for a ballpark number. Event 24: ```text Valohai offers straightforward, predictable pricing to help you concentrate on innovation rather than expenses. With Valohai, you pay per user license, not the scale of work you do on the platform, so you can scale your AI projects without worrying about hidden costs or unexpected fees. Learn more ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/20e915d8-e33b-47a5-9eed-64b54aadb4e4) · [Read transcript](https://www.ax-check.com/valohai.com/sessions/qwen.json) Pricing was a generic, unsourced description (platform fee + BYO cloud, quote-based) with no cost figure, plan, or workload assumptions; pricing page never checked (seq 141) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No self-service credential acquisition occurred. The agent never obtained a real VALOHAI_TOKEN or VALOHAI_PROJECT during the session; it only probed the live API with a dummy/invalid token, which correctly returned a 403 authentication_failed response. This confirms the API endpoint is reachable but does not demonstrate an authenticated operation. The agent explicitly states it needs the user to supply credentials to proceed. Event 121: ```text Auth failed (403) — check VALOHAI_TOKEN. Body: {"message":"Invalid token.","code":"authentication_failed"} ``` Event 97: ```text --- body of /api/projects/ with dummy token --- ``` Event 141: ```text actually *launching* a run needs credentials I don't have — `VALOHAI_TOKEN` and a `VALOHAI_PROJECT` UUID ``` #### Hallucinated URLs - **Guessed unversioned API paths that 404'd before finding the real base**: The agent initially wrote create_execution.py against https://app.valohai.com/api (no version) and then probed invented paths like /api/v1/projects/ and a separate api.valohai.com host, all of which failed (404 or connection failure) before it discovered the real /api/v0/ base via a redirect. These were guessed paths that did not work, though the agent self-corrected using observed redirect evidence rather than leaving them in place. Event 97: ```text 404 /api/projects/ ``` Event 102: ```text 000 https://api.valohai.com/api/projects/ ``` Event 102: ```text 404 https://app.valohai.com/api/v1/ ``` #### Blockers - **No real Valohai credentials available in the sandbox**: The session had no VALOHAI_TOKEN or VALOHAI_PROJECT value supplied, which is a normal credential/login requirement rather than a product defect. The agent's create_execution.py correctly refuses to run without them, and the agent clearly flagged this as the boundary where it stopped, rather than fabricating a fake success. Event 92: ```text Set VALOHAI_TOKEN and VALOHAI_PROJECT env vars first. exit=1 ``` Event 141: ```text I can't fire a real execution or read back its logs/metrics without you supplying a token ``` - **valohai-yaml SDK's actual API diverged from assumed interface**: The agent assumed a valohai_yaml.build() convenience function existed, which does not exist in the installed SDK version (0.61.0); this is an agent knowledge/assumption error rather than a product defect, since the agent was able to discover and use the correct object-based Config/Step/Parameter API through introspection. Event 38: ```text AttributeError: module 'valohai_yaml' has no attribute 'build' ``` Event 68: ```text yaml.representer.RepresenterError: ('cannot represent an object', OrderedDict(...)) ``` #### Suggested Changes - **Document the real valohai-yaml SDK object API in quickstart docs**: Add a code sample showing Config/Step/Parameter construction plus serialize()+lint().is_valid() (and the need for a custom YAML dumper for OrderedDict output) to the official Valohai SDK quickstart or README, since the agent had to reverse-engineer this via Python introspection after a documented-sounding build() function did not exist. Verify by having a new developer generate valohai.yaml from the doc snippet without hitting AttributeError or RepresenterError. Event 38: ```text AttributeError: module 'valohai_yaml' has no attribute 'build' ``` Event 73: ```text yaml.representer.RepresenterError: ('cannot represent an object', OrderedDict({'name': 'train' ``` - **Clarify the REST API base path in docs**: State explicitly in the API reference that the versioned base path is https://app.valohai.com/api/v0/ and that unversioned or alternate paths like /api/v1/ or a separate api.valohai.com host return 404/connection failures. Verify by confirming a fresh GET to /api/v0/projects/ with a valid token returns project data on the first attempt without needing to follow a redirect manually. Event 97: ```text 404 /api/projects/ ``` Event 101: ```text location: /api/v0/ ``` ### Task given to each agent Help me build a simple example using Valohai. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: C · 55/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Homepage answers Markdown requests ```text Homepage returned text/html for a text/markdown request; no Markdown representation offered. ``` - **Failed** — llms.txt provides an actionable documentation index ```text https://valohai.com/llms.txt returns HTTP 404, so no documentation index exists. ``` - **Skipped** — llms.txt provides navigation guidance ```text No llms.txt body exists (404), so navigation guidance cannot be assessed. ``` - **Skipped** — llms.txt mentions offered API, MCP, and skills ```text No llms.txt body exists (404), so API/MCP/skills mentions cannot be assessed. ``` - **Pass** — A compact guide representation exists ```text Docs pages offer Markdown via .md suffix; migrate-with-ai-skills.md is a standalone Markdown guide. ``` - **Pass** — A focused guide is directly retrievable ```text migrate-with-ai-skills.md fetched directly as text/markdown with actionable migration steps. ``` - **Pass** — Equivalent instructions fit a token budget ```text Guide is 1471 tokens, well under 8000 and below 40% of HTML extraction. ``` - **Skipped** — Product-docs links survive format changes ```text Homepage Markdown unsupported, so link preservation across formats cannot be measured. ``` - **Pass** — The compact guide is independently actionable ```text Migration guide gives concrete npx skills install, agent prompts, and vh CLI commands. ``` - **Pass** — Install and next-step links resolve ```text Getting-started and migration pages fetched successfully; docs links resolve. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Getting Started links a Quickstart to run your first execution in 5 minutes. ``` - **Skipped** — Installation commands are extractable ```text Fetched pages show no installation commands; only a quickstart link is offered. ``` - **Skipped** — Code examples are available without interaction ```text Fetched pages contain no code examples, only prose and a resource table. ``` - **Skipped** — Prerequisites and auth boundaries are explicit ```text Fetched pages do not state prerequisites or API key/auth boundaries. ``` ### Pricing - **Failed** — Prices are stated, not gated ```text Pricing page states per-user license but shows no actual price; FAQ admits pricing is not shown. ``` - **Failed** — Pricing units and limits are explicit ```text Pricing page gives per-user model but no numeric rate, seat minimums, or usage limits. ``` - **Failed** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 1 fell short. DeepSeek V4.1 Flash: Gave no made-up number. Based on the pricing page, it said pricing is a fixed per-user monthly fee by custom quote with no public price list, platform usage is unlimited, and compute is billed separately. (seq 38, seq 124, seq 126) Kimi K3: Gave pricing grounded in the fetched page: per-user license, custom quote with no public figure, free trial, compute paid separately; did not invent a number (seq 24, seq 55) Qwen 3.8 Max: Pricing was a generic, unsourced description (platform fee + BYO cloud, quote-based) with no cost figure, plan, or workload assumptions; pricing page never checked (seq 141) This behavioural item does not affect the fast grade. ``` - **Pass** — Pricing is readable without interaction ```text Pricing page renders per-user license model and FAQ in plain HTML, no interaction needed. ``` ### Activation - **Skipped** — An API reference or OpenAPI spec is reachable ```text No API reference or OpenAPI spec page was fetched; docs mention REST API only in passing. ``` - **Skipped** — An MCP server is documented and well-formed ```text No MCP server documentation appears in any fetched Valohai page. ``` - **Pass** — A CLI install path is documented ```text Docs list a CLI section and skills page shows vh commands like vh lint and vh execution run. ``` - **Pass** — SDK packages resolve on their registries ```text PyPI registry lookup for valohai-llm returned HTTP 200. ``` - **Pass** — Agent skills are published ```text Migration page documents open-source Valohai Agent Skills installable via npx skills add valohai/valohai-skills. ``` [Full report data](https://www.ax-check.com/valohai.com/report.json)