# AX Check: pulumi.com
Checked 2026-09-29.

Pulumi's docs, install steps, and pricing are all clear upfront.
All 23 checked items passed, including install, API/MCP docs, and pricing. Every plan price is stated plainly with no login required.

## End-to-end onboarding not demonstrated

## Coding sessions
All three independent sessions completed the task and correctly reported Pulumi's pricing straight from the live pricing page: Free, Essentials $40/mo, Pro $400/mo, and Enterprise $2,000/mo, with consistent notes on credits, overage, and metering assumptions.

### DeepSeek V4.1 Flash
[View public run](https://agents.withgauge.com/p/runs/45b4a222-f57c-4116-a0c6-842be2159c53) · [Read transcript](https://www.ax-check.com/pulumi.com/sessions/deepseek.json)
Final output gives concrete tier prices (Free/Essentials $40/Pro $400/Enterprise $2000) sourced from a live fetch of pulumi.com/pricing (seq 11/17), with credit/overage/metering assumptions spelled out.
#### End-to-end onboarding
- **End-to-end onboarding not demonstrated**: Agent never obtained real Pulumi Cloud credentials (no PULUMI_ACCESS_TOKEN was acquired) and never performed an authenticated operation against the hosted Pulumi Cloud product. It only wrote a local Node.js program and tested it entirely offline using Pulumi's in-process mock layer (pulumi.runtime.setMocks), which explicitly bypasses the real engine, cloud credentials, and any hosted backend. This is a local mock, which the standard explicitly excludes from verification.
  Event 56:

  ```text
  process.env.PULUMI_CONFIG = JSON.stringify({ "pulumi-example:prefix": "demo" });
  
  pulumi.runtime.setMocks({
  ```
  Event 66:

  ```text
  I have no Pulumi token (`PULUMI_ACCESS_TOKEN` unset), no CLI installed, and no cloud credentials, and running a live `pulumi up` is exactly the long-running, credential-dependent operation you told me to avoid.
  ```
  Event 60:

  ```text
  # tests 2
  # pass 2
  # fail 0
  ```

#### Hallucinated URLs
None identified in this transcript.

#### Blockers
- **No Pulumi CLI or cloud credentials available for live deploy**: The real Pulumi quickstart requires installing the Pulumi CLI, authenticating to Pulumi Cloud (or another backend), and running `pulumi up`, which provisions real billable cloud resources. This sandbox had no Pulumi CLI, no PULUMI_ACCESS_TOKEN, and no AWS/cloud credentials, and the seed prompt explicitly told the agent not to run long-running commands. This is a test-environment limitation combined with the prompt's own restriction, not a product defect — needing a normal login/token is expected for a hosted SaaS product.
  Event 66:

  ```text
  I have no Pulumi token (`PULUMI_ACCESS_TOKEN` unset), no CLI installed, and no cloud credentials, and running a live `pulumi up` is exactly the long-running, credential-dependent operation you told me to avoid.
  ```
  Event 6:

  ```text
  /usr/local/bin/node
  /usr/local/bin/python3
  /usr/local/go/bin/go
  ---
  ---
  
  
  Command exited with code 1
  ```
- **Pulumi Output object not directly awaitable in tests**: Agent error, quickly self-corrected: initial test code awaited the exported Output values directly, which returned the internal OutputImpl object instead of the resolved value, causing two test failures. This was a misunderstanding of the SDK's Output API, not a product defect, and was fixed within the same turn by calling `.promise()`.
  Event 48:

  ```text
  Expected values to be strictly equal:
      + actual - expected
      
      + OutputImpl {
      +   __pulumiOutput: true,
  ```
  Event 57:

  ```text
  const petName = await infra.petName.promise();
  ```
  Event 60:

  ```text
  # tests 2
  # pass 2
  # fail 0
  ```

#### Suggested Changes
- **Add a working example of unwrapping Output values in unit tests**: On the unit testing docs page (https://www.pulumi.com/docs/iac/using-pulumi/testing/unit/), include a complete, runnable snippet showing that Output values returned from a program module must be unwrapped via `.promise()` rather than awaited directly. The agent's first test attempt awaited the Output object directly and got the internal OutputImpl structure instead of the resolved value, only succeeding after switching to `.promise()`. Verify by having a first-time user copy the snippet and run `node --test` without hitting the same assertion failure.
  Event 48:

  ```text
  Expected values to be strictly equal:
      + actual - expected
      
      + OutputImpl {
      +   __pulumiOutput: true,
  ```
  Event 57:

  ```text
  const petName = await infra.petName.promise();
  ```

### Kimi K3
[View public run](https://agents.withgauge.com/p/runs/45b3eee3-f690-40cf-ac50-682b2636d169) · [Read transcript](https://www.ax-check.com/pulumi.com/sessions/kimi.json)
Final output states CLI/SDK are free/open-source, self-managed state is free, Pulumi Cloud is priced per resource-under-management with free/team/enterprise tiers, and flags 'exact numbers change; check pulumi.com/pricing' as the assumption caveat.
#### End-to-end onboarding
- **End-to-end onboarding not demonstrated**: No Pulumi Cloud account or access token was obtained or used. The agent explicitly ran pulumi login --local (a local file-based backend), never authenticated against the hosted Pulumi Cloud service, and stated no PULUMI_ACCESS_TOKEN existed. The 'random' provider resources created are local-state only and not a hosted product operation.
  Event 6:

  ```text
  PULUMI_ACCESS_TOKEN set: 
  
  ```
  Event 43:

  ```text
  Logged in to freestyle-vm as root (file://~)
  Created stack 'dev'
  ```
  Event 61:

  ```text
  No `PULUMI_ACCESS_TOKEN` in the environment, so I couldn't demo the **hosted Pulumi Cloud** workflow — I used the local state backend instead. If you want the SaaS features (web console, shared state, secrets), you'd need to create a free account and token.
  ```

#### Hallucinated URLs
None identified in this transcript.

#### Blockers
- **No Pulumi Cloud token available for hosted workflow**: The sandbox environment had no PULUMI_ACCESS_TOKEN, so the agent could not authenticate to the actual hosted Pulumi Cloud service. This is a missing-credentials limitation of the test environment, not a product defect; the agent worked around it by using a local file-based state backend instead, which does not demonstrate the hosted product.
  Event 6:

  ```text
  PULUMI_ACCESS_TOKEN set: 
  
  ```
  Event 61:

  ```text
  No `PULUMI_ACCESS_TOKEN` in the environment, so I couldn't demo the **hosted Pulumi Cloud** workflow — I used the local state backend instead.
  ```
- **Slow sandbox network stalled CLI install**: The official install script (get.pulumi.com) timed out after 120 seconds, and a direct tarball download also timed out mid-transfer before succeeding on a resumed retry. This is a test-environment network limitation, not a product defect, and the agent recovered by resuming the partial download.
  Event 13:

  ```text
  Command timed out after 120 seconds
  ```
  Event 22:

  ```text
  curl: (28) Operation timed out after 30000 milliseconds with 53156544 out of 96427204 bytes received
  DOWNLOAD FAILED/BLOCKED
  ```
- **Pulumi CLI ran program without SDK installed**: After scaffolding the Python project and running pulumi up, the run failed because the pulumi Python module wasn't installed in a virtualenv the CLI could find. This was agent error (skipped venv setup and virtualenv runtime option) rather than a product defect, and was resolved by creating a venv, installing requirements, and adding a virtualenv option to Pulumi.yaml.
  Event 43:

  ```text
  ModuleNotFoundError: No module named 'pulumi'
      pulumi:pulumi:Stack pulumi-example-dev  It looks like the Pulumi SDK has not been installed. Have you run pip install?
  ```

#### Suggested Changes
- **Bundle a pinned CLI release with pre-baked venv handling for Python runtime**: The pinned CLI version (3.142.0) obtained via the tarball did not auto-create a virtualenv for the Python runtime, requiring a manual python3 -m venv step and a Pulumi.yaml edit to add 'options: virtualenv: venv'. Verify by running pulumi up immediately after 'pulumi new python' style scaffolding on this same CLI version and confirming it succeeds without manual venv setup or Pulumi.yaml edits.
  Event 43:

  ```text
  It looks like the Pulumi SDK has not been installed. Have you run pip install?
      If you are running in a virtualenv, you must run pip install -r requirements.txt from inside the virtualenv.
  ```
  Event 45:

  ```text
  This CLI version needs the venv set up explicitly:
  ```

### Qwen 3.8 Max
[View public run](https://agents.withgauge.com/p/runs/53190046-0401-4686-988f-5f86756d2999) · [Read transcript](https://www.ax-check.com/pulumi.com/sessions/qwen.json)
Final output gives a tier table pulled live from pulumi.com/pricing (seq 58-59 curl fetch) with explicit assumptions: CLI is free/self-hosted state used here, Cloud tiers with credit-based overage, resource counts per tier.
#### End-to-end onboarding
- **End-to-end onboarding not demonstrated**: The agent never obtained or used real Pulumi Cloud credentials. It deliberately avoided cloud/product accounts, deploying only against a local file-based state backend with the random and tls providers (which run entirely locally, no cloud API calls). No Pulumi Cloud login, API token, or hosted state backend was used at any point, so there's no authenticated operation against the actual hosted product to verify.
  Event 35:

  ```text
  Now deploying against a **local file-based state backend** (no Pulumi Cloud account, no login):
  ```
  Event 95:

  ```text
  === STATE FILES ===
  .pulumi-state/.pulumi/meta.yaml
  .pulumi-state/.pulumi/history/pulumi-demo/dev/dev-1790670207884996960.history.json.attrs
  ```
  Event 106:

  ```text
  Cloud-targeted stacks need credentials. I couldn't demo an AWS/GCP deploy — no credentials in this environment.
  ```

#### Hallucinated URLs
None identified in this transcript.

#### Blockers
- **No cloud credentials available to test a real hosted deploy**: The sandbox had no AWS/Azure/GCP or Pulumi Cloud credentials, so the agent could not exercise a real cloud deployment or Pulumi Cloud login. This is a test-environment limitation (missing credentials), not a product defect — the agent worked around it by using local-only providers (random, tls) to demonstrate the CLI workflow.
  Event 10:

  ```text
  Empty repo, network is up, but no Pulumi CLI. Pulumi does have a real developer workflow (SDK + CLI), so I'll proceed. Let me check installability.
  ```
  Event 106:

  ```text
  Cloud-targeted stacks need credentials. I couldn't demo an AWS/GCP deploy — no credentials in this environment.
  ```
- **Primary GitHub release download failed, required fallback URL**: The Pulumi installer's first download attempt from github.com timed out with a curl error; the installer script's own built-in fallback to get.pulumi.com/releases succeeded automatically. This was transient network/product installer behavior, not an agent error, and it self-recovered without intervention.
  Event 20:

  ```text
  curl: (56) Failure when receiving data from the peer
  + Error encountered, falling back to https://get.pulumi.com/releases/sdk/pulumi-v3.265.0-linux-x64.tar.gz...
  ```
- **file:// backend path ambiguity caused a stack pointing to the wrong home directory**: Using PULUMI_BACKEND_URL=file://~ resolved '~' against the OS account home (/root) rather than $HOME (/sandbox), so initial state ended up in an unexpected location. This is a Pulumi CLI behavior interacting with the sandbox's HOME override; the agent diagnosed it and fixed it by pinning an explicit repo-local path.
  Event 54:

  ```text
  HOME=/sandbox
  /root/.pulumi/stacks/pulumi-demo/dev.json
  ```
  Event 68:

  ```text
  Fixing one ambiguity: `file://~` resolved to `/root` (not `$HOME=/sandbox`). Making state repo-local and explicit instead:
  ```
- **CLI refused to create missing file-backend directory**: After switching to an explicit repo-local file backend path, 'pulumi stack init' and 'pulumi up' failed because the CLI does not auto-create the backend root directory. The agent fixed this by adding a mkdir -p step to its wrapper script before invoking pulumi.
  Event 82:

  ```text
  error: unable to open state directory "file:///sandbox/repo/.pulumi-state": stat /sandbox/repo/.pulumi-state: no such file or directory
  ```
  Event 84:

  ```text
  Needs the directory to pre-exist — fixing the wrapper:
  ```

#### Suggested Changes
- **Document that file:// backend paths must pre-exist**: In the Pulumi CLI docs for the local/file state backend (e.g. the backends reference used when running 'pulumi stack init' with PULUMI_BACKEND_URL=file://...), note explicitly that the target directory is not auto-created and must exist before use. Verify by running 'pulumi stack init' against a fresh non-existent file:// path and confirming the docs' guidance prevents the 'unable to open state directory' error observed here.
  Event 82:

  ```text
  error: unable to open state directory "file:///sandbox/repo/.pulumi-state": stat /sandbox/repo/.pulumi-state: no such file or directory
  ```
- **Clarify file://~ home-directory expansion behavior in backend docs**: In the file-based state backend documentation, clarify that '~' in a file:// URL expands using the OS account's home directory, not the $HOME environment variable, since this diverged unexpectedly in a sandboxed/containerized environment with a custom HOME. Verify by testing 'pulumi login file://~' with HOME set to a non-default path and confirming the docs match observed resolution.
  Event 54:

  ```text
  HOME=/sandbox
  /root/.pulumi/stacks/pulumi-demo/dev.json
  ```
  Event 68:

  ```text
  Fixing one ambiguity: `file://~` resolved to `/root` (not `$HOME=/sandbox`). Making state repo-local and explicit instead:
  ```

### Task given to each agent
Help me build a simple example using Pulumi. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop.

No product credentials were supplied and no purchases were authorized.

## Score: A · 100/100 (provisional)
Grades come from completed site checks. Coding sessions and skipped checks do not affect the score.

### Clarity
- **Pass** — Homepage answers Markdown requests

  ```text
  Homepage returned text/markdown (790 tokens) when requested with Accept: text/markdown.
  ```

- **Pass** — llms.txt provides an actionable documentation index

  ```text
  llms.txt links docs, install, get-started, registry, and agent endpoints with examples.
  ```

- **Pass** — llms.txt provides navigation guidance

  ```text
  llms.txt organizes site and docs sections with descriptions and starting guidance for agents.
  ```

- **Pass** — llms.txt mentions offered API, MCP, and skills

  ```text
  llms.txt links the Cloud REST API, MCP server, and Agent Skills pages.
  ```

- **Pass** — A compact guide representation exists

  ```text
  Homepage serves text/markdown via content negotiation; docs pages also offer .md twins.
  ```

- **Pass** — A focused guide is directly retrievable

  ```text
  llms.txt and docs pages fetched as markdown with concrete commands and endpoints.
  ```

- **Pass** — Equivalent instructions fit a token budget

  ```text
  Homepage markdown is 790 tokens vs 11401 HTML, well under budget.
  ```

- **Pass** — Product-docs links survive format changes

  ```text
  Homepage markdown retains docs, llms.txt, sitemap and product links.
  ```

- **Pass** — The compact guide is independently actionable

  ```text
  llms.txt gives agents concrete endpoints, commands and auth for API, registry, MCP and skills.
  ```

- **Pass** — Install and next-step links resolve

  ```text
  Sampled install and next-step links (/docs/install/, /docs/get-started/, signup) returned 200.
  ```


### Onboarding
- **Pass** — Docs lead to a relevant quickstart

  ```text
  llms.txt links Get Started tutorial: install Pulumi, write first program, deploy to AWS/Azure/GCP.
  ```

- **Pass** — Installation commands are extractable

  ```text
  llms.txt links Download & Install page; CLI runnable via npx pulumi with concrete commands.
  ```

- **Pass** — Code examples are available without interaction

  ```text
  llms.txt and CLI docs show inline code examples (pulumi api, pulumi do) without interaction.
  ```

- **Pass** — Prerequisites and auth boundaries are explicit

  ```text
  CLI API docs state auth reuses pulumi login token or PULUMI_ACCESS_TOKEN; exit code 3 on auth failure.
  ```


### Pricing
- **Pass** — Pricing is readable without interaction

  ```text
  Pulumi pricing page renders as static Markdown with all plan tiers and rates visible.
  ```

- **Pass** — Prices are stated, not gated

  ```text
  Free $0, Essentials $40/mo, Pro $400/mo, Enterprise $2,000/mo all stated plainly.
  ```

- **Pass** — Pricing units and limits are explicit

  ```text
  Units explicit: per-resource hourly rates, credits, workflow minutes, Neo tokens per million.
  ```

- **Pass** — Agents identify pricing and its assumptions

  ```text
  3 of 3 sessions were judged on pricing; 0 fell short. DeepSeek V4.1 Flash: Final output gives concrete tier prices (Free/Essentials $40/Pro $400/Enterprise $2000) sourced from a live fetch of pulumi.com/pricing (seq 11/17), with credit/overage/metering assumptions spelled out. Kimi K3: Final output states CLI/SDK are free/open-source, self-managed state is free, Pulumi Cloud is priced per resource-under-management with free/team/enterprise tiers, and flags 'exact numbers change; check pulumi.com/pricing' as the assumption caveat. Qwen 3.8 Max: Final output gives a tier table pulled live from pulumi.com/pricing (seq 58-59 curl fetch) with explicit assumptions: CLI is free/self-hosted state used here, Cloud tiers with credit-based overage, resource counts per tier. This behavioural item does not affect the fast grade.
  ```


### Activation
- **Pass** — An API reference or OpenAPI spec is reachable

  ```text
  Pulumi Cloud REST API reference and OpenAPI spec are documented and reachable.
  ```

- **Pass** — An MCP server is documented and well-formed

  ```text
  MCP server documented with hosted URL, OAuth auth, tools, and per-assistant config.
  ```

- **Pass** — A CLI install path is documented

  ```text
  CLI install path documented via Download & Install and npx pulumi usage.
  ```

- **Pass** — SDK packages resolve on their registries

  ```text
  Registry API returns published Pulumi packages with versions and schemas.
  ```

- **Pass** — Agent skills are published

  ```text
  Agent Skills published on agentskills.io standard with install instructions.
  ```



[Full report data](https://www.ax-check.com/pulumi.com/report.json)