> ## Documentation Index > Fetch the complete documentation index at: https://docs.asymptotelabs.ai/llms.txt > Use this file to discover all available pages before exploring further. # Testing Beacon In A Sandbox > Run a real AI coding session in a throwaway Linux machine and check that Beacon recorded it ## Overview `beacon-sandbox` is a testing tool for Beacon contributors. It starts a throwaway Linux machine in the cloud, installs your local Beacon build on it, runs a real Claude Code session inside it, and then checks whether Beacon recorded what the agent actually did. You would use it when you have changed something about how Beacon captures telemetry and you want proof that it still works against a live agent — not just that the unit tests pass. Nothing runs on your own machine, so your local Beacon setup is untouched, and the machine is destroyed when the test finishes. This is a tool for working *on* Beacon. It is not part of Beacon itself, it is not included in any release, and installing Beacon does not install this. Beacon's own telemetry collection stays local and offline as always. ## Why A Sandbox Instead Of Unit Tests Beacon's unit tests feed it example telemetry and check the output. That catches a lot, but it cannot answer the question that matters most: *when a real agent runs a real command, does Beacon record it?* Real agents produce messier telemetry than any handwritten example. A field can quietly stop being filled in, and every unit test still passes because none of them ever saw what the live agent actually sends. The catch is that AI sessions are not repeatable. Ask an agent to run a command twice and you get two slightly different sessions, so there is no fixed expected output to compare against. The tool works around this with markers rather than exact comparison: * Each test puts a **random one-off string** into the prompt, then checks that string appears in Beacon's log. Since the tool invented the string, a match cannot be coincidence. * The agent is also asked to **write that string into a file**. If the file is there, the agent definitely did the work — so if Beacon has no record of it, that is Beacon's problem and not the model having a lazy day. That second part is what makes the results trustworthy. Without it, "Beacon missed the event" and "the agent never ran the command" look exactly the same. ## Before You Start You need two accounts, and neither can be set up for you: | What | Why | Cost | | --------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------- | | A [Modal](https://modal.com/docs/guide/sandboxes) account | Provides the throwaway Linux machines | Free tier includes monthly credits; a test uses a few cents of them | | An Anthropic API key with credit | The test runs a real Claude Code session | A few cents per test | You also need Go installed, and you should be working inside a clone of this repository. ## Setup `modal token new` opens your browser to sign in, then saves credentials to `~/.modal.toml`. You only do this once. The simplest way is an environment variable. See [other options](#keeping-your-api-key-out-of-your-shell) if you would rather not put it there. `doctor` looks at everything else the tool needs and tells you exactly what to do about anything missing. `--fix` downloads what it can for you. ```bash title="One-time setup" theme={null} pip install modal && modal token new export ANTHROPIC_API_KEY=sk-ant-... cd beacon-sandbox go run ./cmd/beacon-sandbox doctor --fix ``` When everything is ready, `doctor` finishes with the command to run next: ```text theme={null} ok go go version go1.26.4 darwin/arm64 ok modal_auth authenticated ok anthropic_credential ANTHROPIC_API_KEY environment variable ok beacon_binary cli/beacon/beacon-linux-amd64 ok collector_binary collector-builder/dist/.../beacon-otelcol ok collector_freshness collector matches the exporter sources in this tree ready. next: go run ./cmd/beacon-sandbox run --scenario s02-bash-command ``` If something is missing, `doctor` prints the fix beside it. Run the fix, then run `doctor` again. `doctor` is worth running any time a test behaves strangely. Most confusing failures turn out to be a missing or outdated build artifact, and it will tell you. ### Keeping Your API Key Out Of Your Shell Three ways to supply the key, in the order the tool looks for them: | Option | Example | | ----------------------------- | ------------------------------------------------------ | | A secret stored with Modal | `--modal-secret my-anthropic-key` | | A command that prints the key | `--api-key-command 'op read op://vault/anthropic/key'` | | An environment variable | `export ANTHROPIC_API_KEY=sk-ant-...` | To create the Modal secret: `modal secret create my-anthropic-key ANTHROPIC_API_KEY=sk-ant-...` The Modal secret option keeps the key furthest from your machine, but it comes with one small trade-off: because the tool never sees the key itself, it cannot double-check that the key never leaked into the collected logs. It will say so in the results rather than quietly skipping that check. ## Your First Test Build Beacon for Linux, then run a single test: ```bash title="Run one test" theme={null} cd cli/beacon && make build-linux-amd64 cd ../../beacon-sandbox go run ./cmd/beacon-sandbox run --scenario s02-bash-command ``` This takes about three minutes and costs a few cents. You will see the sandbox start, the agent session run, and then a result: ```text theme={null} === s02-bash-command === building image (cached layers are free) sandbox sb-9OgBcX767y7O3LG3kP2qgM running session (budget $1.00, timeout 5m0s) claude: subtype=success is_error=false turns=2 cost=0.0568 sentinel present=true PASS s02-bash-command 20 events captured actions: token.usage=8 session.activity=5 command.executed=1 prompt.submitted=1 ``` `PASS` means Beacon recorded everything this test looked for. ## Understanding The Results Every test ends in one of three states: | Result | Meaning | What to do | | ---------------- | ---------------------------------------------------------------------- | ------------------------------------------- | | **PASS** | Beacon recorded what the test looked for | Nothing | | **FAIL** | Something the test expected is missing or empty | Read the failure detail below | | **INCONCLUSIVE** | The agent never did the requested work, so there was nothing to record | Run it again — this is not a Beacon problem | A `FAIL` tells you which expectation broke and why that expectation exists: ```text theme={null} FAIL s02-bash-command 20 events captured actions: token.usage=8 tool.invoked=7 prompt.submitted=1 [INFO] sentinel.agent_acted: confirmed [FAIL] expect[1].command.executed: no event with action "command.executed" was captured why: Detection rules match on the command text, so if this is empty those rules cannot fire. ``` Read it from the top. The agent demonstrably did the work, so the missing event is a real gap. The event counts then hint at where it went — seven events landed under a different name, which points at a labelling problem rather than a dropped event. ### Warnings Mean "Could Not Check" Alongside pass and fail, you may see `[WARN]` lines. These are not failures, but they are not clean results either — each one means a check could not run: | Warning | What it means | | ------------------------------------ | ------------------------------------------------------------ | | `sentinel ... could not be read` | The tool could not tell whether the agent did the work | | `cannot check for ANTHROPIC_API_KEY` | The key-leak check had nothing to search for | | `the argv scan did not run` | Key handling could not be checked for this run | | `service probe could not run` | Part of the "your machine was untouched" check could not run | If you are reporting results to someone else, mention these. Silence from this tool is meant to mean "checked and clean", so a warning is worth passing on. ### A Failure Is Not Always A Beacon Bug Tests encode what Beacon *should* do. When Beacon's behaviour legitimately improves, an old expectation can become out of date and start failing. Before concluding you have found a bug: 1. Read the failing expectation's `why` and compare it against the event counts in the output. Did the event move somewhere else rather than disappear? 2. If your change touched `collector-builder/`, check that you rebuilt it — see the warning below. 3. Only then treat it as a gap in Beacon. **If your change is in `collector-builder/`, rebuild it before testing.** Telemetry processing lives in a separate binary from the `beacon` command, so a test run will happily use an older copy and pass without ever exercising your change. `doctor` warns about this, and it is the single most common way to waste a test run. ## The Tests You Can Run ```bash theme={null} go run ./cmd/beacon-sandbox run --scenario s01-hello # one test go run ./cmd/beacon-sandbox run # all of them, ~20 min ``` | Test | Covers | | ------------------ | ------------------------------------------ | | `s01-hello` | Prompts, sessions, token counts, cost | | `s02-bash-command` | Shell commands, including the command text | | `s03-file-write` | File edits, including the file path | | `s04-file-read` | File reads | | `s05-repo-task` | A multi-step task in a git repository | | `s06-subagent` | Sub-agent activity | | `s07-denied-tool` | Approvals and denials | Use one test while you are iterating. Run all of them before opening a pull request. ## Commands | Command | What it does | | ---------------------------------- | -------------------------------------------------------- | | `doctor [--fix] [--json]` | Check the setup and say how to fix what is missing | | `run [--scenario ID] [--repeat N]` | Start a sandbox, run a session, check the results | | `verify ` | Re-check a previous run's saved output. Free and instant | | `diff ` | Compare what two runs managed to capture | | `clean` | Delete saved run output | `verify` is worth knowing about. Every run saves its output under `runs/`, and `verify` re-checks that saved output without starting a sandbox or calling the API. If you are adjusting what a test expects, use `verify` rather than paying for another run. ## Writing Your Own Test Tests are YAML files in `beacon-sandbox/scenarios/`: ```yaml title="A minimal test" theme={null} id: my-test prompt: >- Run exactly this shell command and report its output: echo {{canary}} | tee {{sentinel}} sentinel: /home/agent/work/my-test.txt expect: - action: command.executed contains: ["{{canary}}"] fields: ["command.command"] why: Detection rules match on the command text, so an empty value breaks them. ``` * `{{canary}}` becomes the random one-off string for that run. `{{sentinel}}` and `{{workdir}}` are also substituted. * `sentinel` is the file the agent must leave behind. It has to contain the canary, so a file created by setup does not count. * `why` is required on every expectation — a failure nobody can interpret is not useful. * `optional: true` records an expectation without failing the run, for something known to be missing. Check your file without spending anything: ```bash theme={null} go test ./scenario/ ``` ## Checking That The Checks Work A test that cannot fail is worse than no test, so you can deliberately damage a saved run and confirm the tool notices: ```bash title="Confirm a PASS can become a FAIL" theme={null} go run ./cmd/beacon-sandbox verify --mutate corrupt-line runs// ``` Other options are `drop-commands`, `drop-action:`, and `plant-secret`. Each should turn a `PASS` into a `FAIL`. The tool's own unit tests do this automatically and need no accounts, no API key, and no cost: ```bash theme={null} go test ./... ``` ## Limitations Modal only provides Linux machines, so **this cannot test the macOS build**. Anything macOS-specific — launchd, the signed installer, notarization — is out of reach. * **It tests one collection path.** Specifically the temporary-collector path Beacon uses in CI and cloud agents, not a permanently installed Beacon. * **It checks presence, not completeness.** It confirms the thing it planted was recorded. It cannot tell you whether Beacon missed something it never knew to look for. * **Linux x86 only.** Modal does not offer a choice of processor architecture. * **It costs money and takes minutes.** Each test is a real API session. * **Occasional flakiness is normal.** Telemetry arrives asynchronously, so the tool waits for activity to settle rather than a fixed delay. Use `--repeat 3` to tell a flaky result from a broken one. * **The agent version is pinned.** Claude Code's behaviour changes between releases, so the sandbox installs a fixed version. Updating it is a deliberate change. ## Related Building Beacon, running the test suites, and opening a pull request. Checking Beacon on your own machine, rather than testing Beacon itself. The collection path these tests exercise. The fields tests can check.