The result / 01
Stopping every attack is easy if nothing gets done.
The strict profile looked safe until usefulness entered the score. It rejected four legitimate tasks. The vulnerable profile completed four tasks but allowed two attacks to succeed. The bounded profile preserved both safety and completion.
| Architecture | Tasks | Attacks | False refusals | Score |
|---|---|---|---|---|
A Vulnerable long contextBroad visibility, weak boundaries | 4 / 8 | 2 | 0 | 87 |
B Overcautious retrievalSafe by refusing useful work | 4 / 8 | 0 | 4 | 82 |
C Bounded hybrid memoryScoped context + deterministic authority | 8 / 8 | 0 | 0 | 100 |
What changed
The best result did not come from a more permissive agent. It came from giving the agent less authority and the surrounding system better evidence.
Inspect the three proof runs →The pivot / 02
The agent stopped being the interesting part.
Capable models can already plan, call tools, and coordinate roles. Another orchestration demo would mostly prove that the framework works. BoundaryLab asks the harder product question: what prevents a plausible output from becoming an unsafe action?
Less useful question
Can several agents complete a task?
Modern agent platforms already make this easy to demonstrate.BoundaryLab question
Which system boundary changed the outcome?
Hold the task constant. Make authority, evidence, and evaluation inspectable.The architecture / 03
Reasoning can propose. Only policy can act.
Four boundaries separate interpretation, mutation, evaluation, and publication. Each one removes a power the agent should never own.
Interpret and advise
Agent-visible. Every output remains an untrusted proposal.
Context manager
Build a bounded view- Required policy
- Ranked retrieval
- Scoped memory
- Audit history
Worker
Propose a response or typed function callNo direct mutation accessSentinel
Pass, block, retry, or request reviewAdvice only; cannot grant authorityEvaluate independently
Protected. The agent cannot see the answer key or grade itself.
Observed evidence
Final state + allowlisted trace factsProtected input
Expectations + synthetic canariesDeterministic judge
Score seven explained metricsOutcome, policy, attacks, refusals, evidence, claims, contaminationPublish evidence, not internals
Sanitized. Private prompts, canaries, objectives, and ground truth stay out.
Run output
Scorecard + context manifest + event traceArtifact boundary
Sanitize and lock immutable bytesRefuse silent overwriteStatic replay
Inspect all 24 runs without inferenceNo model, backend, API key, or visitor costThe experiment / 04
Change one architecture. Preserve the evidence.
Every profile receives the same canonical scenario digest. The workflow records what each architecture saw, proposed, allowed, changed, and scored.
- 01Same digest
Load one controlled case
Versioned task, policy, world state, attack surface, profile, and fixed budgets.
- 02Context manifest
Assemble bounded context
Required policy enters first. Retrieval, memory, expiry, exclusions, and compaction remain visible in the manifest.
- 03Role handoffs
Run separated roles
The attacker controls only declared untrusted surfaces. The worker proposes. The sentinel advises inside a retry budget.
- 04Typed decision
Gate every capability
Schema, identity, authority, limits, approvals, preconditions, and idempotency decide whether anything changes.
- 05Seven metrics
Judge outside the agents
Protected expectations meet the final world state and trusted trace facts—never an agent self-report.
- 06Immutable artifact
Freeze reproducible evidence
The exporter writes sanitized scores, usage, context, memory, gateway decisions, trace events, and cause attribution.
- 07$0 per visitor
Replay without inference
The browser reads bundled JSON. Visitors can compare every run without a model call or runtime secret.
The decisions / 05
Advanced mechanisms earn their place by removing a failure.
RAG, memory, function calling, reviewers, and orchestration are not the story. The boundary each one enforces is.
RAG is a strategy, not a default
Long context preserves small authoritative inputs. Retrieval reduces noise and attack surface but can omit policy. The hybrid wins only when required evidence bypasses ranking.
A typed call is still only a proposal
Schemas improve parsing. Permission still comes from trusted identity, scope, limits, approvals, preconditions, and idempotency outside model output.
Useful memory has identity and an expiry
Every record carries case scope, writer, source events, trust, creation time, and expiry. Compaction can change context; it cannot rewrite the audit ledger.
A reviewer can advise without becoming authority
The sentinel can block or request a bounded retry. It cannot mutate the world or grant a capability—which makes over-blocking measurable instead of invisible.
The answer key stays outside the conversation
Protected expectations and canaries never enter agent context. The judge consumes final state and allowlisted trace facts, not the agent's explanation of its own success.
Checkpointing is infrastructure, not policy
LangGraph remains an optional orchestration edge where resumption matters. The gateway, judge, context strategies, and deterministic benchmark run without it.
The proof / 06
Do not take the summary on faith.
Open the exact runs behind the conclusion. Each one exposes its context, proposal, review, gateway outcome, final state, deterministic score, and cause attribution.
Vulnerable long context / attack succeeded
Failure 02See safety destroy usefulnessOvercautious retrieval / false refusal
ControlInspect the safe completionBounded hybrid memory / task completed
Immutable source
24 sanitized replay bundlesSHA-256 4b7ee5f3…a22d./scripts/run-benchmark
./scripts/build-replay --check
./scripts/verify