Testing Beacon In A Sandbox

Overview

beacon-sandbox is a testing tool for Beacon contributors. It starts a throwaway Linux machine in the cloud, installs your local Beacon build on it, runs a real Claude Code session inside it, and then checks whether Beacon recorded what the agent actually did. You would use it when you have changed something about how Beacon captures telemetry and you want proof that it still works against a live agent — not just that the unit tests pass. Nothing runs on your own machine, so your local Beacon setup is untouched, and the machine is destroyed when the test finishes.

This is a tool for working on Beacon. It is not part of Beacon itself, it is not included in any release, and installing Beacon does not install this. Beacon’s own telemetry collection stays local and offline as always.

Why A Sandbox Instead Of Unit Tests

Beacon’s unit tests feed it example telemetry and check the output. That catches a lot, but it cannot answer the question that matters most: when a real agent runs a real command, does Beacon record it? Real agents produce messier telemetry than any handwritten example. A field can quietly stop being filled in, and every unit test still passes because none of them ever saw what the live agent actually sends. The catch is that AI sessions are not repeatable. Ask an agent to run a command twice and you get two slightly different sessions, so there is no fixed expected output to compare against. The tool works around this with markers rather than exact comparison:

Each test puts a random one-off string into the prompt, then checks that string appears in Beacon’s log. Since the tool invented the string, a match cannot be coincidence.
The agent is also asked to write that string into a file. If the file is there, the agent definitely did the work — so if Beacon has no record of it, that is Beacon’s problem and not the model having a lazy day.

That second part is what makes the results trustworthy. Without it, “Beacon missed the event” and “the agent never ran the command” look exactly the same.

Before You Start

You need two accounts, and neither can be set up for you:

What	Why	Cost
A Modal account	Provides the throwaway Linux machines	Free tier includes monthly credits; a test uses a few cents of them
An Anthropic API key with credit	The test runs a real Claude Code session	A few cents per test

You also need Go installed, and you should be working inside a clone of this repository.

Setup

Install and sign in to Modal

modal token new opens your browser to sign in, then saves credentials to ~/.modal.toml. You only do this once.

Provide your Anthropic API key

The simplest way is an environment variable. See other options if you would rather not put it there.

Run the setup check

doctor looks at everything else the tool needs and tells you exactly what to do about anything missing. --fix downloads what it can for you.

One-time setup

pip install modal && modal token new
export ANTHROPIC_API_KEY=sk-ant-...

cd beacon-sandbox
go run ./cmd/beacon-sandbox doctor --fix

When everything is ready, doctor finishes with the command to run next:

ok    go                     go version go1.26.4 darwin/arm64
ok    modal_auth             authenticated
ok    anthropic_credential   ANTHROPIC_API_KEY environment variable
ok    beacon_binary          cli/beacon/beacon-linux-amd64
ok    collector_binary       collector-builder/dist/.../beacon-otelcol
ok    collector_freshness    collector matches the exporter sources in this tree

ready. next: go run ./cmd/beacon-sandbox run --scenario s02-bash-command

If something is missing, doctor prints the fix beside it. Run the fix, then run doctor again.

doctor is worth running any time a test behaves strangely. Most confusing failures turn out to be a missing or outdated build artifact, and it will tell you.

Keeping Your API Key Out Of Your Shell

Three ways to supply the key, in the order the tool looks for them:

Option	Example
A secret stored with Modal	`--modal-secret my-anthropic-key`
A command that prints the key	`--api-key-command 'op read op://vault/anthropic/key'`
An environment variable	`export ANTHROPIC_API_KEY=sk-ant-...`

To create the Modal secret: modal secret create my-anthropic-key ANTHROPIC_API_KEY=sk-ant-...

The Modal secret option keeps the key furthest from your machine, but it comes with one small trade-off: because the tool never sees the key itself, it cannot double-check that the key never leaked into the collected logs. It will say so in the results rather than quietly skipping that check.

Your First Test

Build Beacon for Linux, then run a single test:

Run one test

cd cli/beacon && make build-linux-amd64
cd ../../beacon-sandbox
go run ./cmd/beacon-sandbox run --scenario s02-bash-command

This takes about three minutes and costs a few cents. You will see the sandbox start, the agent session run, and then a result:

=== s02-bash-command ===
  building image (cached layers are free)
  sandbox sb-9OgBcX767y7O3LG3kP2qgM
  running session (budget $1.00, timeout 5m0s)
  claude: subtype=success is_error=false turns=2 cost=0.0568
  sentinel present=true
  PASS  s02-bash-command
    20 events captured
    actions: token.usage=8 session.activity=5 command.executed=1 prompt.submitted=1

PASS means Beacon recorded everything this test looked for.

Understanding The Results

Every test ends in one of three states:

Result	Meaning	What to do
PASS	Beacon recorded what the test looked for	Nothing
FAIL	Something the test expected is missing or empty	Read the failure detail below
INCONCLUSIVE	The agent never did the requested work, so there was nothing to record	Run it again — this is not a Beacon problem

A FAIL tells you which expectation broke and why that expectation exists:

FAIL  s02-bash-command
  20 events captured
  actions: token.usage=8 tool.invoked=7 prompt.submitted=1
  [INFO] sentinel.agent_acted: confirmed
  [FAIL] expect[1].command.executed: no event with action "command.executed" was captured
        why: Detection rules match on the command text, so if this is empty those
             rules cannot fire.

Read it from the top. The agent demonstrably did the work, so the missing event is a real gap. The event counts then hint at where it went — seven events landed under a different name, which points at a labelling problem rather than a dropped event.

Warnings Mean “Could Not Check”

Alongside pass and fail, you may see [WARN] lines. These are not failures, but they are not clean results either — each one means a check could not run:

Warning	What it means
`sentinel ... could not be read`	The tool could not tell whether the agent did the work
`cannot check for ANTHROPIC_API_KEY`	The key-leak check had nothing to search for
`the argv scan did not run`	Key handling could not be checked for this run
`service probe could not run`	Part of the “your machine was untouched” check could not run

If you are reporting results to someone else, mention these. Silence from this tool is meant to mean “checked and clean”, so a warning is worth passing on.

A Failure Is Not Always A Beacon Bug

Tests encode what Beacon should do. When Beacon’s behaviour legitimately improves, an old expectation can become out of date and start failing. Before concluding you have found a bug:

Read the failing expectation’s why and compare it against the event counts in the output. Did the event move somewhere else rather than disappear?
If your change touched collector-builder/, check that you rebuilt it — see the warning below.
Only then treat it as a gap in Beacon.

If your change is in collector-builder/, rebuild it before testing. Telemetry processing lives in a separate binary from the beacon command, so a test run will happily use an older copy and pass without ever exercising your change. doctor warns about this, and it is the single most common way to waste a test run.

The Tests You Can Run

go run ./cmd/beacon-sandbox run --scenario s01-hello   # one test
go run ./cmd/beacon-sandbox run                        # all of them, ~20 min

Test	Covers
`s01-hello`	Prompts, sessions, token counts, cost
`s02-bash-command`	Shell commands, including the command text
`s03-file-write`	File edits, including the file path
`s04-file-read`	File reads
`s05-repo-task`	A multi-step task in a git repository
`s06-subagent`	Sub-agent activity
`s07-denied-tool`	Approvals and denials

Use one test while you are iterating. Run all of them before opening a pull request.

Commands

Command	What it does
`doctor [--fix] [--json]`	Check the setup and say how to fix what is missing
`run [--scenario ID] [--repeat N]`	Start a sandbox, run a session, check the results
`verify <run-dir>`	Re-check a previous run’s saved output. Free and instant
`diff <before> <after>`	Compare what two runs managed to capture
`clean`	Delete saved run output

verify is worth knowing about. Every run saves its output under runs/, and verify re-checks that saved output without starting a sandbox or calling the API. If you are adjusting what a test expects, use verify rather than paying for another run.

Writing Your Own Test

Tests are YAML files in beacon-sandbox/scenarios/:

A minimal test

id: my-test
prompt: >-
  Run exactly this shell command and report its output:
  echo {{canary}} | tee {{sentinel}}
sentinel: /home/agent/work/my-test.txt
expect:
  - action: command.executed
    contains: ["{{canary}}"]
    fields: ["command.command"]
    why: Detection rules match on the command text, so an empty value breaks them.

{{canary}} becomes the random one-off string for that run. {{sentinel}} and {{workdir}} are also substituted.
sentinel is the file the agent must leave behind. It has to contain the canary, so a file created by setup does not count.
why is required on every expectation — a failure nobody can interpret is not useful.
optional: true records an expectation without failing the run, for something known to be missing.

Check your file without spending anything:

go test ./scenario/

Checking That The Checks Work

A test that cannot fail is worse than no test, so you can deliberately damage a saved run and confirm the tool notices:

Confirm a PASS can become a FAIL

go run ./cmd/beacon-sandbox verify --mutate corrupt-line runs/<run-dir>/

Other options are drop-commands, drop-action:<action>, and plant-secret. Each should turn a PASS into a FAIL. The tool’s own unit tests do this automatically and need no accounts, no API key, and no cost:

go test ./...

Limitations

Modal only provides Linux machines, so this cannot test the macOS build. Anything macOS-specific — launchd, the signed installer, notarization — is out of reach.

It tests one collection path. Specifically the temporary-collector path Beacon uses in CI and cloud agents, not a permanently installed Beacon.
It checks presence, not completeness. It confirms the thing it planted was recorded. It cannot tell you whether Beacon missed something it never knew to look for.
Linux x86 only. Modal does not offer a choice of processor architecture.
It costs money and takes minutes. Each test is a real API session.
Occasional flakiness is normal. Telemetry arrives asynchronously, so the tool waits for activity to settle rather than a fixed delay. Use --repeat 3 to tell a flaky result from a broken one.
The agent version is pinned. Claude Code’s behaviour changes between releases, so the sandbox installs a fixed version. Updating it is a deliberate change.

Contributing

Building Beacon, running the test suites, and opening a pull request.

Validate a local install

Checking Beacon on your own machine, rather than testing Beacon itself.

beacon ci exec

The collection path these tests exercise.

Event schema

The fields tests can check.

Get Started

Deploy Asymptote

How Asymptote Works

Manage Asymptote

Agent Harness Integrations

For Security & IT Teams

Contributing

Testing Beacon In A Sandbox

Overview

Why A Sandbox Instead Of Unit Tests

Before You Start

Setup

Keeping Your API Key Out Of Your Shell

Your First Test

Understanding The Results

Warnings Mean “Could Not Check”

A Failure Is Not Always A Beacon Bug

The Tests You Can Run

Commands

Writing Your Own Test

Checking That The Checks Work

Limitations

Contributing

Validate a local install

beacon ci exec

Event schema

​Overview

​Why A Sandbox Instead Of Unit Tests

​Before You Start

​Setup

​Keeping Your API Key Out Of Your Shell

​Your First Test

​Understanding The Results

​Warnings Mean “Could Not Check”

​A Failure Is Not Always A Beacon Bug

​The Tests You Can Run

​Commands

​Writing Your Own Test

​Checking That The Checks Work

​Limitations

​Related

Contributing

Validate a local install

beacon ci exec

Event schema

Overview

Why A Sandbox Instead Of Unit Tests

Before You Start

Setup

Keeping Your API Key Out Of Your Shell

Your First Test

Understanding The Results

Warnings Mean “Could Not Check”

A Failure Is Not Always A Beacon Bug

The Tests You Can Run

Commands

Writing Your Own Test

Checking That The Checks Work

Limitations

Related