# AX Check: withwillow.ai Checked 2026-10-23. Willow's docs, pricing and setup are clear and agent-ready. 22 of 23 checks passed; only a broken setup-guide link needs fixing. All 3 agent sessions found and explained pricing correctly. ## Onboarding needs a login ## Coding sessions All three independent sessions completed their task. DeepSeek and Kimi both read the pricing page and accurately reported the Free plan at $0 with its user and integration limits, plus Enterprise as contact-sales. Qwen misidentified Willow as an open-source PyPI library rather than the hosted product, concluding no price applied, but still flagged that assumption explicitly. ### DeepSeek V4.1 Flash [View public run](https://agents.withgauge.com/p/runs/91ab14bc-827c-4e11-a484-a0a3fb7888c8) · [Read transcript](https://www.ax-check.com/withwillow.ai/sessions/deepseek.json) Quoted the Free plan at $0/mo with its limits (5 users, 5 integrations, proxy MCP only). Said Premium and Enterprise are contact-sales only and there is no sandbox. All of this came from the billing docs it fetched. (seq 32, seq 59, willow-example/README.md) #### End-to-end onboarding - **Onboarding needs a login**: The agent only reached the public, unauthenticated catalog endpoint (GET /api/mcp-servers with no token), which returned 820 public connectors. It never obtained a real Willow API token or machine-user credential — signup requires an interactive browser flow with a work email, which the agent could not complete in the sandbox. Both authenticated code paths it wrote (org and gateway modes) exited immediately with BLOCKED messages because WILLOW_TOKEN / WILLOW_ACCESS_KEY / WILLOW_SECRET were never set, so no authenticated operation against the product was ever demonstrated. Event 57: ```text BLOCKED: WILLOW_TOKEN is not set. Create a free org at https://app.withwillow.ai/signup, then Admin > API Tokens > Generate Token (scope admin:read) and export WILLOW_TOKEN=wxt_... ``` Event 57: ```text BLOCKED: Set WILLOW_ORG, WILLOW_ACCESS_KEY and WILLOW_SECRET (Manage > Users > Machine Users). ``` Event 38: ```text --- token search --- (none above = no local willow creds) ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-service credential provisioning in sandbox**: Willow signup requires an interactive browser OAuth/email flow with a work email (personal providers like Gmail rejected) to create an organization and mint an API token or machine-user key. This is a normal product login requirement, not a product defect, but it stopped the agent from running any authenticated example (org MCP-server listing or gateway tool calls) since no human supplied credentials during the session. Event 18: ```text Go to app.withwillow.ai/signup and sign up with Google, GitHub, or email. Your first sign-in creates a new organization on the Free plan and makes you its owner. ``` Event 57: ```text BLOCKED: WILLOW_TOKEN is not set. Create a free org at https://app.withwillow.ai/signup, then Admin > API Tokens > Generate Token (scope admin:read) and export WILLOW_TOKEN=wxt_... ``` #### Suggested Changes - **Publish an official SDK package instead of relying on OpenAPI codegen**: The docs page for 'Try the Willow API' states Willow ships no first-party SDK and directs developers to generate one from the OpenAPI specs. Verify by checking the 'Official packages' section of docs.withwillow.ai/docs/developers/try-the-api — it lists only the CLI and a local MCP bridge, no API client library. A published Admin/User API SDK (npm/PyPI) would remove the codegen step for a first-time integration. Event 32: ```text Willow does not publish official SDK packages. The APIs are described by OpenAPI 3 specs, which you can feed to any OpenAPI client generator ``` - **Offer a non-interactive way to provision a trial token**: The quickstart's only path to a usable API token is the interactive dashboard signup and click-through token generation, with no CLI or API-based trial provisioning. Check docs.withwillow.ai/docs/developers/try-the-api — Steps 1-2 are both manual dashboard actions. Adding a scripted/CLI trial-token issuance path would let an automated agent complete the full example without a human in the loop. Event 18: ```text In the admin dashboard, go to Admin > API Tokens . Select Generate Token . ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/7a373edd-8f10-4838-b512-d71654abca5a) · [Read transcript](https://www.ax-check.com/withwillow.ai/sessions/kimi.json) Quoted Free at $0 with its limits: up to 15 users, no card, Admin/User API tokens. It said Enterprise is contact-sales and listed what that plan adds. Both came from the fetched pricing doc. (seq 38, seq 58) #### End-to-end onboarding - **Onboarding needs a login**: Agent could not self-serve credentials for the actual Willow product (Admin/User API, org gateway, CLI). Signup requires a work email and interactive browser OAuth, and the wxt_ bearer token must be generated by hand in the admin dashboard — none of which the agent could do unattended in the sandbox, and no credentials were supplied. The only operation demonstrated was against the public, unauthenticated docs MCP server (search_docs/get_doc), which the docs explicitly describe as requiring no authentication, so it does not count as an authenticated product operation. Event 32: ```text Start at [app.withwillow.ai/signup](https://app.withwillow.ai/signup). To try the API, see [Try the Willow API for free](https://docs.withwillow.ai/docs/developers/try-the-api.md). ``` Event 38: ```text Go to [app.withwillow.ai/signup](https://app.withwillow.ai/signup) and sign up with Google, GitHub, or email. Your first sign-in creates a new organization on the Free plan and makes you its owner. ``` Event 48: ```text "name":"search_docs","title":"Search Willow docs","description":"Search the Willow documentation. Returns matching pages with their URL and a short snippet." ``` Event 58: ```text I could not go past the public docs server into the actual product APIs (`/api/mcp-servers`, the org gateway, the CLI sign-in) — all require an authenticated organization. If you give me a `wxt_` token (Free plan, `admin:read` scope is enough to start), I can immediately exercise the Admin API ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No self-serve path to a real API token without human sign-up**: The actual Willow Admin/User API requires a work-email signup with interactive OAuth (Google/GitHub/email) in a browser and manual generation of a wxt_ bearer token in the admin dashboard. This is a normal authentication requirement of the product, not a defect, but it stopped the agent from exercising anything beyond the public unauthenticated docs server. Event 38: ```text Go to [app.withwillow.ai/signup](https://app.withwillow.ai/signup) and sign up with Google, GitHub, or email. Your first sign-in creates a new organization on the Free plan and makes you its owner. ``` Event 58: ```text bootstrap is human-gated. Sign-up requires a **work email** (Gmail/Yahoo rejected) and interactive OAuth in a browser, and the `wxt_` token is generated in the admin dashboard — none of which I can do from this sandbox, and no credentials were provided. ``` #### Suggested Changes - **Offer a scripted or CLI-based token bootstrap for the Free plan**: On the 'Try the Willow API for Free' page (docs.withwillow.ai/docs/developers/try-the-api.md), the only way to get a wxt_ token is interactive dashboard sign-up and manual click-through. Adding a non-interactive path (e.g., a CLI command or API endpoint to provision a Free-plan org and token from a script) would let automated agents validate the Admin/User API without a human in the loop. Verify by having an agent run the new command and successfully call /api/mcp-servers with the resulting token. Event 38: ```text In the admin dashboard, go to **Admin** > **API Tokens**. 2. Select **Generate Token**. ``` Event 58: ```text none of which I can do from this sandbox, and no credentials were provided ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/6d78b3b8-cfb1-41d7-80bd-fc8298d52b43) · [Read transcript](https://www.ax-check.com/withwillow.ai/sessions/qwen.json) Stated $0 cost based on Willow being an open-source PyPI library (not hosted), and flagged the assumption that a hosted SaaS 'Willow' could not be priced (seq 25, seq 88) #### End-to-end onboarding - **End-to-end onboarding not demonstrated**: No hosted product credentials were ever obtained or used. The agent identified 'Willow' as an open-source Python image library (PyPI package, BSD license, no API/accounts/billing) rather than a hosted SaaS product, installed it locally via pip, and ran a local script. No authentication step occurred because none exists for this library -- there was nothing to onboard to. Event 86: ```text Loaded output/source.png: size=(800, 600) After resize: (400, 300) After crop: (300, 200) Wrote output/resized.png and output/cropped.jpg ``` Event 88: ```text I have no credentials, documentation, or verified access to any hosted "Willow" service in this environment, and I won't fabricate its pricing or API details. ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No hosted Willow product exists to act on**: The agent researched the name 'Willow' and found only a local open-source Python imaging library (combining Pillow/Wand/OpenCV) with no SDK, API, pricing, or account system. This is an ambiguity/product-identification issue rather than a defect in any actual product: the seed prompt assumed a hosted product named Willow that could not be located. The agent correctly flagged this instead of fabricating details. Event 25: ```text Willow 1.12.0 A Python image library that sits on top of Pillow, Wand and OpenCV ``` Event 88: ```text **There is no pricing.** Willow is free, open-source software (BSD license, on PyPI). It's not a hosted product — there's no API, no accounts, no usage-based billing. ``` - **Library import crashes without Django configured**: Importing willow.image raised a Django ImproperlyConfigured error because Django happened to be installed in the sandbox and Willow probes Django settings on import even when Django is unrelated to the use case. This is a product (library) behavior quirk, not a credential or environment limitation; the agent worked around it by calling settings.configure() before import. Event 39: ```text django.core.exceptions.ImproperlyConfigured: Requested setting WILLOW_OPTIMIZERS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. ``` - **Several API mismatches with Willow's documented-sounding interface**: The agent's initial assumptions about Willow's constructor, mode attribute, and save_as_png/save_as_jpeg signatures were wrong and caused repeated runtime errors (TypeError, AttributeError, UnrecognisedOperationError) before converging on the correct open()/save_as_png(f) usage. This reflects agent trial-and-error against the real library API rather than a documented example, and was fully resolved within the same session. Event 49: ```text TypeError: Image() takes no arguments ``` Event 70: ```text AttributeError: 'PNGImageFile' object has no attribute 'mode' ``` Event 78: ```text TypeError: PillowImage.save_as_png() missing 1 required positional argument: 'f' ``` #### Suggested Changes - **Disambiguate the Willow name on the project's PyPI/docs landing page**: Multiple unrelated products share the name 'Willow' (the agent noted Willow Voice, WillowTree, willow.io HR as other possibilities). On the PyPI page or willow.wagtail.org homepage, add a clear one-line description distinguishing this as the Wagtail/Torchbox open-source Python imaging library, not a hosted SaaS, to prevent confusion for developers searching generically. Verify by checking that the PyPI summary and docs homepage explicitly state it is a local library with no hosted API. Event 25: ```text A Python image library that sits on top of Pillow, Wand and OpenCV ``` Event 88: ```text there are several products with that name (Willow Voice, WillowTree, willow.io HR, etc.) ``` - **Guard the Django settings probe in willow/__init__.py setup()**: When Django is present in the environment but not configured or used, importing willow currently raises ImproperlyConfigured instead of a clear, catchable warning. In willow/registry.py register_optimizer (and the setup() call chain in willow/__init__.py), wrap the settings access so it degrades gracefully (e.g., try/except around getattr(settings, 'WILLOW_OPTIMIZERS', False)) when Django settings are unconfigured. Verify by importing willow in an environment with Django installed but settings.configure() never called, and confirming no exception is raised. Event 39: ```text File "/opt/freestyle/python/lib/python3.12/site-packages/willow/registry.py", line 97, in register_optimizer enabled_optimizers = getattr(settings, "WILLOW_OPTIMIZERS", False) ``` - **Document the Image.open(f) pattern and required save_as_X(f) file-argument signature in the quickstart**: The agent's first attempts used Image(bytes) as a constructor and called save_as_png()/save_as_jpeg() with no arguments, both of which fail against the real API (Image.open(file_obj) and save_as_png(file_obj) are required). Add a minimal, complete code snippet to the Willow quickstart (willow.wagtail.org) showing open() and save_as_png(f)/save_as_jpeg(f) with a file-like object. Verify by running the documented snippet verbatim and confirming no TypeError occurs. Event 49: ```text TypeError: Image() takes no arguments ``` Event 78: ```text TypeError: PillowImage.save_as_png() missing 1 required positional argument: 'f' ``` ### Task given to each agent Help me build a simple example using Willow. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 98/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Failed** — Install and next-step links resolve ```text Onboarding links Per-Client Setup Guides, but that page returns 404. ``` - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown with 200 when requested with Accept: text/markdown. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt lists recommended pages: platform, identity, tools, governance, marketplace, pricing, docs. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt groups content under About, Capabilities, Recommended Pages and Optional sections. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt notes API-to-MCP, MCP gateway, and GitHub open-source; docs MCP and skills exist. ``` - **Pass** — A compact guide representation exists ```text Docs pages served as Markdown, e.g. onboarding.md and connect-mcp.md, plus llms.txt indexes. ``` - **Pass** — A focused guide is directly retrievable ```text Onboarding guide retrievable as Markdown with concrete 4-step first-tool workflow. ``` - **Pass** — Equivalent instructions fit a token budget ```text Onboarding guide is 665 tokens, well under the 8000-token budget. ``` - **Pass** — Product-docs links survive format changes ```text Homepage Markdown supported; docs link to docs.withwillow.ai survives in both representations. ``` - **Pass** — The compact guide is independently actionable ```text Onboarding guide gives timed steps: accept invite, connect AI client, run first tool call. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text Docs onboarding page is a concrete quickstart: accept invite, connect client, run first tool. ``` - **Pass** — Installation commands are extractable ```text CLI reference gives extractable install: npm install -g @willowai/cli, then willow init and auth login. ``` - **Pass** — Code examples are available without interaction ```text CLI reference shows inline code examples for install, init, tools call, and config. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Prerequisites (invite email, AI client) and auth (OAuth login, API token scopes) are explicit. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders plan tiers, feature tables and FAQs as static text without interaction. ``` - **Pass** — Prices are stated, not gated ```text Free plan stated as $0 forever; Enterprise is contact-sales with full feature list. ``` - **Pass** — Pricing units and limits are explicit ```text Free tier limits explicit: up to 15 users, 15 integrations, proxy-only MCP. ``` - **Pass** — Agents identify pricing and its assumptions ```text 3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Quoted the Free plan at $0/mo with its limits (5 users, 5 integrations, proxy MCP only). Said Premium and Enterprise are contact-sales only and there is no sandbox. All of this came from the billing docs it fetched. (seq 32, seq 59, willow-example/README.md) Kimi K3: Quoted Free at $0 with its limits: up to 15 users, no card, Admin/User API tokens. It said Enterprise is contact-sales and listed what that plan adds. Both came from the fetched pricing doc. (seq 38, seq 58) Qwen 3.8 Max: Stated $0 cost based on Willow being an open-source PyPI library (not hosted), and flagged the assumption that a hosted SaaS 'Willow' could not be priced (seq 25, seq 88) This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Admin, User and SCIM OpenAPI specs fetched successfully with auth details. ``` - **Pass** — An MCP server is documented and well-formed ```text Public read-only docs MCP server documented with URL, tools and scopes. ``` - **Pass** — A CLI install path is documented ```text Willow CLI install documented: npm install -g @willowai/cli, init, auth login. ``` - **Pass** — SDK packages resolve on their registries ```text npm registry lookup for @willowai/cli returned HTTP 200. ``` - **Pass** — Agent skills are published ```text Willow skill published at .well-known/agent-skills/willow-docs/SKILL.md, installable via CLI. ``` [Full report data](https://www.ax-check.com/withwillow.ai/report.json)