How Hurricane Sentinel Works

Summary

Sentinel is a local agent harness for Linux. A supervisor runs each agent as a Leash-leashed user with model calls pointed at a Muzzle listener, so enforcement is genuine: Leash gates the agent's file/network/command access per-UID and Muzzle inspects its model I/O. Runs are durable and checkpointed, and a flagged action pauses the run for a human approval before it proceeds.

The whole control plane (supervisor, web, API, cron) runs as the leashed user, not root — a worker is a same-user subprocess of it. The one operation that still needs root — changing Leash egress policy — is brokered by a tiny root sentinel-broker service over a root:<leashed> socket with two schema-validated verbs and no arbitrary execution; it is the only Sentinel service that runs as root.

The supervised run

1. Supervisor (root). The sentinel service owns run lifecycle and the approval queue. For each run it drops privileges and spawns a worker process as the leashed user, with the worker's model endpoint set to a Muzzle listener.
2. Run engine (durable + checkpointed). A run is one agent task. Its state — message history, step, status — is persisted to the leashed user's ~/.sentinel/runs/<id>/. Status moves running → waiting_approval → done | failed, and every step (and every pause) is checkpointed so a run resumes exactly where it stopped. A run's directory is reserved exclusively before anything is written to it, and the id is a full UUID rather than a short prefix of one: a run must never be able to land on top of another run's state or append its work to another run's audit history. Run directories created before this are shorter and keep working unchanged — nothing reads meaning into the length.
3. Agent worker (as the leashed user). The loop: build messages → call the model through Muzzle → read the model's tool calls → for each, consult the action policy:
   • auto → run the tool (its file/network/command effects are enforced by Leash);
   • needs_approval → checkpoint, enqueue an approval, and pause the run;
   • deny → feed the denial back to the agent. No tool calls → the model's answer is final → the run is done.
4. Async approvals. Pending approvals are persisted. An operator lists them and approves or denies via the CLI; on approval the supervisor resumes the run and the tool executes, then the loop continues.

When a guardrail blocks

Enforcement is only half the story — a block the agent can't understand just makes it fail or loop. So every block is made legible to both the agent and the operator, without adding any new execution path:

• Muzzle blocks (model I/O). A rejected model call is typed by direction and category (secrets / PII / prompt-injection / content-policy). If the block is on the agent's output, the worker feeds it a rewrite note and re-drives the turn (bounded by a per-run retry cap) so it can produce a compliant reply; exhausting the cap fails the run cleanly. If the block is on the input it was handed, the agent can't process it — the run stops with a plain explanation and does not retry.
• Leash / sandbox denials (OS actions). A tool failure is classified: a filesystem denial (Landlock or Leash's fs surface) comes back to the model as the path is outside your workspace — write to your workspace or /tmp, and a blocked network reach comes back as the host isn't on the operator's allow-list — legible guidance, not a raw errno.
• Operator visibility. Each block emits a guardrail_block event that the web console renders as a color-coded card in the run feed (source-tinted for Muzzle vs Leash/sandbox). For a Leash denial the supervisor (leashed) correlates the event with Leash's decision log and attaches the exact reason (e.g. "1.1.1.1:80 not in whitelist") — the correlation is a supervisor concern, done once outside the run, not per worker.
• In-context guidance. When a run is subject to Muzzle and/or the sandbox, a compact guardrails skill is auto-injected into the agent's system prompt (only the parts that apply to that run), teaching it what each block means and how to respond — rewrite, stop-and-explain, or pick an allowed path/host.

This is pure observability layered on top of the existing enforcement: a guardrail_block never executes the blocked action, the injected skill is text only (it grants no tools), and the correlation with Leash's log happens root-side only. Making a block visible never makes it bypassable.

Why files live in the leashed user's home

Per-agent runtime state (souls/, skills/, memory/, runs/) lives under the leashed user's ~/.sentinel, so the worker — which runs as that user under Leash's filesystem enforcement — can read and write its own state without anyone having to open up a privileged system directory.

Two deployment shapes, one runtime

Sentinel's guarantees are Linux kernel features — Landlock, seccomp-BPF, cgroups, nftables, fanotify — so Sentinel runs on Linux. There are two ways to get there:

• Direct install on a Linux host. install.sh creates the leashed user, writes the systemd units, and registers with Leash and Muzzle.
• The macOS desktop app. A Tauri shell around a managed Lima VM (Apple Virtualization, Ubuntu guest). The Mac runs the window, the CLI shims and the browser; every enforcement mechanism runs inside the guest.

These are not two ports. The VM installs Sentinel by fetching the same artifact from the same endpoint and running the same install.sh with the same production profile and leashed user — so the guest is a normal Linux install, and every CLI subcommand behaves identically because it is the same code on the same kernel. The only platform-conditional code in the product is a handful of sys.platform != "linux" guards that make the sandbox modules importable on a developer's Mac; they never fire in a real deployment.

What differs is reach, not behaviour: on macOS the host sentinel has no config (nothing is installed there), so commands go through sentinel-runtime cli <command>, which forwards into the guest. The console is port-forwarded to the Mac's loopback only — the VM is NAT-isolated, so the guest binds 0.0.0.0 internally but nothing is exposed to the LAN.

CLI-runtime agents run with a reduced boundary

A codex or claude-code agent is that CLI: Sentinel hands it the prompt and it drives its own agentic loop, with the agent's granted Sentinel tools bridged in as MCP servers so it can still call them. It remains an ordinary agent — scoped, teamed, workspace-bound.

But because the CLI owns the loop and brings its own sandbox, Landlock and seccomp are skipped and its model traffic never passes through Muzzle. What still applies is the leashed user, Leash's egress enforcement, the per-run cgroup and the workspace as cwd.

That trade is deliberate, and it is not silent. A production or airgapped profile refuses such a run unless the operator sets allow_cli_runtime: true, and every run's security_attestation records cli_runtime and muzzle_inspected: false — so an audit can identify exactly which runs had the weaker boundary rather than inferring it.

Their subscription credential is Sentinel's, not the machine's: stored in the secrets store and materialized into a private HOME for the duration of a single run, so the CLI only ever sees the credential chosen for that agent, and it disappears with the run.

Dependencies

Sentinel requires Muzzle and Leash. The installer verifies (and can install) both, and sentinel init --leash <user> [--muzzle] registers the agent's user with Leash, ensures Leash is running, and checks Muzzle.

Identity: one name for the agent and its user

First-run setup asks for one name (sentinel setup prompts "Name your agent (also its Linux user)"; headless installs default to naomi). That single name becomes both the head agent and the leashed Linux user: the display name is preserved for the agent (e.g. Naomi), and a normalized form (lowercase, [a-z0-9_]) is the OS account (e.g. naomi). There is no generic main agent anymore — init creates the named head agent, points default_agent at it, and writes its starter soul.md. --leash <name> doubles as the name on either command.

Migration: existing deployments keep their current leashed user and main agent until you re-run setup with a name (which creates the new named head agent alongside main — nothing is renamed or deleted).

Who is signed in: accounts, sessions, and roles

The console authenticates people, not just "whoever has the token". An operator signs in with a username and password; the password is verified against a scrypt hash and, on success, the server mints a session token — random, per-user, expiring — that the browser sends on every request exactly like the old bearer token, so the streaming and ticket paths are unchanged. Only the SHA-256 of that token is stored, so the session file can't be read back into a working session.

Every request resolves to an identity before anything else runs, and the role is re-read from the user store at that moment (not taken from the session), so disabling an account or demoting someone takes effect on their next request rather than their next login. Roles gate the surface: auditor is read-only everywhere, operator drives runs and resources, approver additionally resolves approval gates, and admin additionally manages users. Guards that matter in practice are enforced server-side — you cannot demote, disable, or delete the last remaining admin.

The original single shared token still authenticates, mapping to a break-glass admin. That is deliberate: it's how a brand-new install creates its first account, and it keeps existing scripts working. Once named admins exist, it should be rotated and treated as emergency access only.

Roles gate the settings that decide how much review happens, not only the actions. Two kinds of write used to be available to any signed-in operator even though each one quietly removes a control:

• Rewriting the guardrails themselves — pushing a new Muzzle policy or a new Leash egress config — now requires admin. These are the boundaries every run is judged against; an operator who can rewrite them is not bounded by them.
• Turning on automatic tool approval — on a chat, a cron job, or an endpoint — now requires approver or admin, the same authority as resolving an approval gate. Granting a run the standing right to skip the queue is the same decision as approving every call it will ever make, so it is the same role. Driving a run that already has automatic approval (continuing it, steering it, regenerating, answering it) takes that authority too: otherwise the gate would only apply to who turned it on, not to who uses it.

A tightening takes effect on the next tool, not the next run. A run's approval policy lives in its own small control record, written atomically, separate from the worker's state checkpoint. Workers re-read it before every tool gate, and it is overlaid whenever run state is loaded or saved, so a worker that checkpointed while the policy was loose cannot write the loose value back over an operator's change. A tool already dispatched finishes; every gate after the change sees the new policy. A missing record means the policy set at creation still stands; a corrupt or unreadable one falls back to queue, the strictest setting, rather than to whatever was there before.

Self-state: soul, operator model, and memory

The head agent maintains its own context across sessions through three self-state files under ~/.sentinel/, all loaded into the system prompt each session:

• agents/<name>.soul.md — its persona (overrides the profile soul). Edited via the edit_soul tool (auto; each write is backed up to .soul.md.bak).
• user.md — its model of the operator, shared across the head agent. Appended via the update_user tool (auto).
• memory — durable notes via remember / recall (see below).

All four self-state/memory tools run auto (no per-edit approval) so the agent can keep itself current without interrupting the operator.

Agents author their own skills and tools

An agent can extend itself at runtime, not just remember things:

• Skills are markdown. create_skill writes ~/.sentinel/skills/<name>/SKILL.md (frontmatter + body) and attaches it to the authoring agent; it loads into that agent's system prompt on the next turn. Skills carry no execution, so create_skill / list_skills run auto.
• Tools are Python. create_tool writes ~/.sentinel/tools/<name>/{tool.yaml,tool.py} (the code must define run(args) returning a string). The tool then loads into the registry like any built-in, and a call runs the code out-of-process as the leashed user — so it inherits the same Leash egress allowlist and Landlock filesystem sandbox as the worker, plus a CPU-time limit and a wall-clock timeout. The built-in run_command tool runs through the same bounded runner the whole harness is standardizing on: a wall-clock timeout, a whole-process-group kill so a command that forks children can't survive the timeout, a CPU ceiling, and an in-memory output cap. (Hard RSS/memory bounding is a deployment control — a systemd/cgroup MemoryMax; see SECURITY.md.) A failure never crashes the run: a sandbox/Leash denial comes back as a legible guardrail reason (and a guardrail_block event — see When a guardrail blocks), while other failures (timeout, exception) come back as an error: … string fed to the model. create_tool / edit_tool / list_tools / delete_tool run auto — managing a tool is not the same as running it.

Approve-once-then-auto. Running an authored tool is gated by trust keyed on a SHA-256 of its code. The first call to a freshly authored (or freshly edited) tool pauses for approval, showing the operator the code they're about to trust; approving runs it and marks that exact code trusted, so later calls run automatically. Editing the tool clears trust (the code changed), so the next call re-prompts with the new code. Denial does not trust it. This keeps the "agent evolves itself" loop fast while keeping a human in the loop the first time any new code would run.

The approval is bound to the bytes, and those bytes are what run. An approval records the digest of the code as it was when the operator read it, and execution re-verifies against that digest and then hands the verified source itself to the child process, which compiles it from memory rather than re-opening the file. So the window between "a human read this code" and "this code ran" contains no second read of the file — editing tool.py after approval cannot change what executes. A tool whose bytes no longer match its approval is refused with authored code requires fresh review and must be approved again; an approval recorded before digests existed carries none, and is refused the same way rather than trusted by default.

Whether code must be verified is decided from the tool registry, which knows an entry was built from authored code, never from the presence of the tool's metadata file. That distinction is load-bearing: when the check was keyed on tool.yaml, deleting that file made an authored tool look like a built-in and skipped verification entirely, while the registry still ran its code. A guard must not be conditional on a file the attacker can remove.

Authored tools run through the same bounded runner as run_command — wall-clock timeout, whole-process-group kill so forked children cannot outlive it, CPU ceiling, and an output cap. Output that exceeds the cap is now an error, not a silently truncated success: a half-result handed to a model reads exactly like a whole one, and the model cannot tell that the rest is missing.

Persistent agents and orchestration

An agent is a named, on-disk profile under the leashed user's ~/.sentinel/agents/: a soul (persona / system prompt), a bound model chosen from your configured list (routed via Muzzle), an extensible skill set (SKILL.md files injected into its context), and the tools it may call. Run one with sentinel run --agent <name>.

One agent acts as the orchestrator. Given a task it can:

• delegate the task to a persistent specialist it picks from the roster, or
• spawn a temporary subagent — composing a persona, model, and skills on the fly — when no specialist fits.

A delegated or spawned subagent runs as its own durable child run under the same Muzzle

• Leash enforcement (possibly a different model), and returns its result to the parent. Delegation is the approval gate (you approve each delegation, seeing the agent + task + grants); within it the child's granted tools run automatically. Tool grants flow parent → child (least privilege), depth is capped, and a temporary subagent is discarded, saved to the roster, or queued for your approval (ask) per config.

Several agents in one conversation

A conversation is not tied to a single agent. Typing @name hands that turn to another agent inside the same thread: it answers there, attributed, and the agent that normally drives the chat does not take a turn. There is no delegation tool call in between — you addressed someone directly.

Underneath, the conversation is the shared thing and each agent's turn is still its own run. That is deliberate. A run's sandbox — its Landlock paths, seccomp filter, tool grants, model and approval queue — is fixed when the run starts, so letting a second agent speak inside someone else's run would either execute it with that agent's reach or require rebuilding the boundary mid-flight. Instead the mentioned agent gets a real child run in its own scope, and its answer is written back into the transcript. Being named in a conversation never widens what an agent can do, and every @ is as visible, cancellable and auditable as any other run.

A mentioned agent joins the discussion rather than receiving a brief: it reads the conversation so far, with each turn attributed to whoever said it, and knows which other agents are present and what each is for. What it does not get is another agent's raw tool output — those arrive as one-line summaries. That boundary does two jobs at once. It keeps file contents, command output and secret-bearing results inside the sandbox that produced them; and it keeps the context budget on the conversation, since tool results are the bulk of a transcript and replaying them would push the discussion into compaction — leaving the agent knowing less, not more. An agent that needs a file reads it itself, with its own tools.

Who @ can reach follows the run scope: a team chat reaches that team's members, a head-scope chat reaches any agent, and a single-agent chat reaches nobody. The rule is enforced server-side, so it holds for the console, the interactive shell and the agent API alike.

Reliable on small models

Because small local models are inconsistent at tool calling, the harness compensates: a tool-use prompt scaffold tells the model to emit real tool calls; the gateway recovers calls a model emits as plain text; tool arguments are normalized and validated against their schema, with a clear correction fed back so the model retries; and a subagent automatically receives the orchestrator's original request so a terse hand-off never loses detail.

Memory and context compaction

Each agent has its own memory bank under ~/.sentinel/memory/<scope>/ — a flat index.md plus an Obsidian-style vault/ of notes (YAML frontmatter + body + [[links]]). The scope comes from the agent's profile (memory: field): private to the agent by default, or shared when several agents name the same scope. The agent uses two tools: remember(title, content, tags?, links?) writes a bounded note, and recall(query) performs model-agnostic lexical retrieval with stopword filtering, light inflection normalization, title/tag/phrase weighting, and recency tie-breaking. Relevant memories are also surfaced at the start of a task, but are explicitly marked as untrusted and potentially stale context. Knowledge persists across runs without requiring an embedding service. Semantic/embedding recall remains a later enhancement.

Context compaction keeps long runs inside the model's window: when a run's history crosses a (character-estimated, tokenizer-agnostic) threshold, Sentinel summarizes the older middle turns into one running "summary so far" message via a model call, and sends the system prompt (soul + skills) plus the most recent turns verbatim alongside it. The threshold is measured against what a compaction can actually shrink: the leading system prefix (soul, attached skills, recalled memory) is kept verbatim, so it is subtracted from the budget up front and only the rest of the transcript is compared. An agent whose prefix alone exceeds the threshold is never compacted — no summary could bring it under — and the run records one compaction_disabled event with the numbers, which is the cue to raise context.compact_threshold_tokens or lighten the agent.

Compaction shapes what the MODEL is sent. It never rewrites what the run keeps. The transcript on disk stays whole, so the chat still renders every turn and your own words are never replaced by a summary of them. This is a correction: compaction used to save the summarized list over the run's messages, which meant a long conversation could come back as stacked "Summary of earlier conversation" blocks with the operator's real prompts gone for good. A background cost optimization must not rewrite the record.

Keeping that cheap needs a cache rather than a rewrite. The run carries a summary plus the point it covers up to, so a long conversation summarizes again only when its tail outgrows the budget — not on every turn — and each extension feeds the previous summary back in, or the earliest history would quietly fall out. The trade is that state.json is no longer bounded by compaction and every save rewrites the whole transcript; that cost is the right side of the bargain against destroying the record.

The conversation record

Every run keeps an append-only events.jsonl beside its state. It records what the model did — each assistant turn with its content, the tools it called, guardrail blocks, approvals, delegations — and, as of this version, what the operator said.

That second half used to be missing: only the model's side was written down, so a transcript could not be rebuilt from the log at all. Each operator turn now appends a user event carrying the text, a timestamp, and a source naming who authored it — web or shell for a person typing, task for the instruction a run was created with, and delegated for a parent agent writing a child run's task. That last distinction is the point of the field: filing agent-written text as operator input would put words in your mouth in the audit trail.

The record is written before the state save and fsynced, because it is the evidence least worth losing; it goes through the same redactor as every other event, so a credential pasted into a prompt never persists verbatim; and images are recorded as a count rather than inline base64. Machine-authored turns that merely carry the user role — a guardrail note, a loop nudge — are deliberately NOT filed as operator input; each already emits its own event.

Validated today

Built and validated on Linux end-to-end: supervised runs (as the leashed user, model via Muzzle) that pause on a needs_approval action and resume on approval — including with Leash filesystem enforcement live; a main agent delegating to a persistent specialist that runs as a parent-linked child and produces the correct result on a small local model; and an agent remembering a fact in one run and recalling it correctly in a later one. A local web UI is the remaining milestone.

The chat sidebar

Chats are ordered by last activity, not creation: a conversation you replied to a minute ago sits above one started last week and never touched. Any chat can be pinned to hold it at the top regardless. Pinning is deliberately not treated as activity — it would otherwise move the chat within the very ordering the pin exists to sit above.

Runs written before this carry no activity stamp; every reader falls back to the creation time rather than treating them as never-touched and sinking them to the bottom. The filesystem and SQLite listings implement the same order, because those two swap under a listing when the mirror drops and a sidebar that reshuffles on failover is its own bug.

The interactive shell

sentinel chat opens a colorized REPL for talking to an agent. Run it as the leashed user (sudo -u <user> sentinel chat) so the agent's tools execute under that account and write to its workspace. The shell:

• streams live progress (each model call and tool call) and types out the final, Muzzle-cleared answer;
• handles approvals inline — a flagged action prompts y/N right there (the async queue is still there for headless run);
• offers completion menus as you type: / for commands (/help, /agents, /use, /model, /status, /memory, /skills, /new, /quit…), @name to send a task to a specific specialist, and $ to browse and attach skills;
• keeps ↑/↓ history across sessions and colorizes commands as you type.

The same loop that powers headless runs drives the chat, so durable state, memory, delegation, and the small-model safeguards (tool-call dedup, the bundled file-discipline skill) all apply. Agents work from a per-user workspace, so a bare filename like notes.txt lands somewhere the leashed user owns.

Chat sessions are durable and resume by default. A plain sentinel chat auto-resumes the most recent session (with a one-line banner); sentinel chat --new (or /new in the shell) starts fresh. sentinel chats lists saved sessions by id and title, and sentinel restore <id> reopens a specific one. Each session is a durable run, so its history and self-state survive across invocations.

Saved workflows

A workflow is a saved, named pipeline at ~/.sentinel/workflows/<name>.yaml. Each step names an agent and a task; the runner executes the steps in order, and outputs thread into later steps through templating:

• {{input}} — the value passed to workflow run --input,
• {{<step>.output}} — a named earlier step's result,
• {{previous}} — the immediately prior step's result.

Each step runs as its own supervised run with its agent (so Muzzle inspection and Leash enforcement apply, and the step's granted tools run within the workflow's authorization). Trigger a workflow with sentinel workflow run <name> [--input …], the chat's /workflow run <name>, or let an agent invoke the run_workflow(name, input) tool as part of its own work (gated by approval policy, like delegate). Workflows are linear in v1 — no branching or loops yet.

Teams

A team is a leader-led roster with a shared workspace, an authored guidelines.md (prepended to every team-scope run), and an activity log — persisted under ~/.sentinel/teams/<name>/. The head agent delegates a task to a team; that spawns the team's leader as a team:<name>-scoped child run, and the leader in turn delegates to its members (delegate_to_member(s)) — each a scoped, sandboxed, approval-gated child run of its own. The supervisor drives the members concurrently in a bounded pool, joins their results back to the leader, and the leader synthesizes an answer. The boundaries are hard: a member's Landlock sandbox excludes other teams' and other agents' private workspaces, cross-team delegation is only reachable from the head, fan-out is bounded (concurrency cap + a per-round item cap), and a hard-killed child is reaped so the leader never hangs waiting on it.

Model providers (local + external)

Every model call goes through a Muzzle listener. Local Ollama is the default; an external provider (OpenAI today) is added as a keyless model alias whose per-provider listener + API key Sentinel pushes into Muzzle — keys live only in Muzzle, never in Sentinel's model store or its API responses. The worker picks the right gateway from the model's resolved provider and dials the listener on loopback. A Muzzle-less install can point straight at Ollama (no external providers, no scrubbing) for a minimal box.

CLI-runtime agents (Codex / Claude Code)

An agent's model can be a CLI runtime instead of a chat model: pick codex or claude-code in the agent editor's model dropdown (they appear there only when that CLI is installed on the host). Such an agent is driven by the CLI's own agentic loop with its own native tools — shell, file edit, web — rather than Sentinel's model-plus-approval-gate loop: Sentinel renders the agent's soul and the conversation into the CLI's task, runs it in the agent's workspace, and returns the CLI's final message as the turn. The point is to run these coding agents on your Codex / Claude subscription, not on a metered API key — the CLI owns its own subscription session (logged in as the leashed user).

The rest of the agent still means what it always did. Any Sentinel tools you grant the agent are handed to the CLI as MCP servers (Sentinel's tools are already MCP-based), so a CLI-runtime agent gets its native tools plus exactly the Sentinel tools you granted — and those flow through the same grant list as any agent. Skills, memory, and workspace stay meaningful. What changes is enforcement: because the CLI sandboxes its own tool calls, Sentinel does not wrap these runs in its per-run Landlock/seccomp or gate each inner tool call — it relies on the OS boundary it always owns (leashed user, Leash egress, the per-run cgroup, the workspace cwd). The console labels the choice "native tools" and says so. See SECURITY.md ("CLI-runtime agents — delegated trust") for the exact posture.

The CLI's step trace is surfaced, not swallowed: Sentinel parses the CLI's event stream (codex exec --json; claude --output-format stream-json) and emits each reasoning step, tool call, and tool result as a run event as it streams. In the console they render as a collapsed "Agent steps" disclosure above that turn's answer, so you can watch what the agent did — its shell/edit/MCP calls and their output — not just the final reply.

Uploaded tools (the MCP tool host)

Beyond built-in and agent-authored tools, an operator can upload a plain .py file. The root process AST-scans it (never importing it) and lists its functions; the operator picks which become tools, and Sentinel appends a managed marker to those functions. A private, built-in stdio MCP server (sentinel-uploads, seeded automatically) serves the marked functions to agents as mcp__sentinel-uploads__<file>__<func> — approval-gated like any MCP tool. Each call runs in a short-lived, killable subprocess (the leashed host spawns one per call and re-checks the managed marker there before importing), so a timeout terminates the work — a runaway or forking uploaded tool can't keep running in the host after its deadline. The file's third-party requirements are detected and installed on one click (leashed pip into a per-user site dir). These tools travel with portability exports.

Workspace file browser

The web console's Files tab is a Finder-like view of what the agents actually produce. Pick a workspace root — the shared leashed workspace, any agent's private workspace, or a team's shared workspace — then navigate folders with a breadcrumb, preview text/code and images inline, and upload / new-folder / rename / delete / download. It's read-write, so it doubles as a way to hand files to an agent (drop them in its workspace) and take files out.

The security-critical part is containment: every path — read or write — resolves through a real-path jail (fsbrowse.safe_path) that confines it to the selected root. A .., an absolute path, or a symlink that points outward is rejected, because the resolved real path must still live inside the root. The browser runs as the leashed user (the web process), so it can only ever touch what that user already can; the jail additionally stops browsing out of the workspace into the rest of that user's home (state, secrets, configs). Uploads are reduced to a bare filename, and the workspace root itself can't be deleted.

Structured endpoints (validated input + output)

A programmable endpoint (sentinel api, POST /api/<name> with a per-endpoint bearer key) turns a request into a normal supervised run — and can shape both ends with an uploaded Pydantic class. The operator uploads a plain .py of BaseModel classes in the web UI; the root process never imports it — a leashed subprocess introspects it (its classes and each class's fields, declared + computed) and records them.

• Structured output: pick an output class for the endpoint. When the agent's run finishes, a leashed with_structured_output(<class>) call — using the agent's own provider, routed through the Muzzle listener like every model call — coerces the answer into a validated instance and returns it as JSON. A model that can't be forced (or a missing class) yields a clean 502, never a crash.
• Structured input: pick an input class. Each call validates the request input against it in a leashed subprocess; a Pydantic error returns a 400 naming the field, and no run is created. The endpoint's single prompt template then lays the validated data out for the model with %field% tokens — scalars inline, lists/objects as JSON, and computed fields (e.g. a transcript derived from a messages list) for custom formatting. The web UI offers each class's fields as insertable chips and highlights valid vs. unknown %tokens% as you type.

Untrusted uploaded classes execute only in the leashed sandbox — at upload (introspect), and per request (validate + coerce) — never in the root web/api process. Endpoints without a selected structure keep their prior free-text behavior.

Codebases (attaching a whole repository)

A git datasource attaches an entire repository read-only. The operator adds it once; agents get search, read, tree, map and schema tools (plus log when history was fetched), and never a write tool.

sentinel datasource add acme-api git \
    --remote git@gitlab.internal:acme/api.git --ref main
sentinel codebase sync acme-api

The split that makes it safe. Cloning and fetching happen control-plane-side as the sentinel-sync identity — never root, never the leashed worker. The connector that answers an agent's search reads the already-materialized tree and cannot open a socket. The thing that talks to the network doesn't run in the sandbox; the thing that runs in the sandbox doesn't talk to the network.

Where the clone lives, and why. /var/lib/sentinel-codebases/<name>/tree/, owned by sentinel-sync, mode 0755 — deliberately not under the state dir. The naomi scope's sandbox grants the whole state tree read-write, so a clone there would be editable in-sandbox by any naomi-scope run, and that same checkout is what every other agent reads through the tools. Landlock has no deny rule and cannot subtract a grant on a parent from a child, so the boundary is an ordinary POSIX owner: readable at every scope, writable at none. Verified on the runtime VM — the leashed user can read the clone and is refused on append, create and delete, while the same file under the state dir is writable.

Search, with no embedding model and no network. Three layers: ripgrep when present, a repo map, and the agent iterating across separate tools.

• ripgrep is an optimisation, not a dependency. Absent one, a bounded Python walker does the same job. rg's only role is producing the candidate file list — snippet extraction and ranking are the same code either way, so the two paths cannot drift into disagreeing about results. Measured on this repository (12,681 files): 3.8–5.4 s per search on the fallback, milliseconds with rg. Install ripgrep for anything repository-sized.
• The repo map is paths plus their top-level symbols, ranked and budgeted, so an agent can choose where to look instead of guessing filenames. Extraction is regex-based on purpose — tree-sitter is the obvious upgrade and compiled grammar wheels land hardest on the airgapped deployments this is for. The map is cached by commit SHA and rebuilt incrementally: after a fetch, git diff --name-only <old>..<new> names exactly the files to re-map, which is what makes a large repository affordable to keep current.
• Agentic iteration is the actual semantic layer. "Where do we handle retries?" is answered by three cheap probes and a few reads, not by one clever call.

The repository is untrusted input, on two axes. Its text can address the agent reading it, so every result is labelled as repository content — data, not instructions — and flows through Muzzle's prompt-injection detector like any other model input. And real repositories contain committed credentials: search snippets are always masked for credential shapes, and read is masked by default with a per-datasource opt-out, because redacting source can corrupt what the agent is trying to understand and that trade belongs to the operator.

Airgapped is a workflow, not a refusal. On the airgapped profile a network remote is refused at add time, naming the profile — enforced where the datasource is saved, in both the CLI and the console, because a rule checked in one entry point is a suggestion. A local mirror path or a git bundle file is accepted on every profile: git bundle create acme.bundle --all on a connected machine, copy it over, --remote /path/to/acme.bundle.

Sync is a fetch and a hard reset to the pinned ref, never a merge — drift self-heals. A dirty working tree is reported and then reset, never silently repaired, because silently repairing tampering destroys the only evidence that it happened.

Working in a codebase (writable checkouts)

A codebase is read-only. A git workspace is a per-run writable checkout of one, so an agent can edit, build, test and commit — and a reviewable branch comes out.

sentinel workspace add acme-work --type git --codebase acme-api
sentinel agent set backend --workspace acme-work

/var/lib/sentinel-codebases/acme-api/tree/        shared, sync-owned, read-only
<state>/workspaces/acme-work/runs/<run-id>/tree/  per-run, leashed-owned, writable

A checkout needs no network and no credential. It is a local clone of a tree already on disk. The deploy key is used when the codebase is first cloned, and again only if an operator approves a push — never in between — so a compromised run cannot reach the remote no matter what it decides.

Per run, not per workspace. Two concurrent runs sharing one working tree corrupt each other silently: one checks out a branch while the other is mid-edit, git status reports the other's changes as its own, and a commit captures both. The mount resolver gives each run its own tree, so Landlock hands them different writable directories.

The checkout has no remotes at all. A local clone leaves origin pointing at the codebase, and an agent's git push then works — writing into the tree every other agent reads. The remote is not needed (the clone already copied every object), so it is removed, and the diff base is recorded at clone time as refs/sentinel/base. There is nothing to push to from inside, at any permission level.

Hooks are off (core.hooksPath=/dev/null). A repository's own hooks are an execution path belonging to whoever wrote the repository, and git commit should not be a way for them to run code in the sandbox.

Pushing is the one privileged action. Never automatic, never by the agent, never --force, and only to sentinel/<agent>/<run-id> — so agent work always lands somewhere a human expects to find it. The operator sees the branch, the commit count and the diffstat before deciding. A human merges it; Sentinel does not.

Checkouts are reaped by a sweeper, not by the worker's exit path — a worker killed mid-run never runs its own cleanup, and two mechanisms that disagree about a crash are worse than one that handles both. A run that is still live is spared, along with its uncommitted work.

Cross-uid git needs help, and this is where the surprises live. The codebase belongs to sentinel-sync and the checkout is made by the leashed worker, so git refuses both ends as "dubious ownership" and clone --local cannot hardlink across owners. Sentinel runs those calls under a throwaway HOME whose gitconfig names exactly the paths involved, and copies objects instead of linking them when the owners differ. The practical consequence: a checkout costs a real object copy, so --max-checkouts is a disk decision, not a formality.

Portability

Export an agent or a whole team as a secret-free .sentinelpkg (a zip): its skills, authored tools, MCP configs with secrets scrubbed, keyless model aliases, souls, team definition + guidelines, and any uploaded tool files + a recomputed requirements list. Import it into another deployment — collisions resolve by operator-chosen rename with all cross-references rewritten, imported tools arrive untrusted and MCP servers disabled, the reserved uploads host is never overwritten, and fresh workspaces are scaffolded. The receiver supplies their own keys and installs requirements, and the team runs.

Importing from OpenClaw and Hermes

sentinel import openclaw <path> / sentinel import hermes <path> (also More ▾ → Import agent in the console, which takes a zipped home) migrate an existing assistant into native Sentinel objects: agents (model resolved against config.models), skills (SKILL.md + resources, source-only frontmatter stripped, resources path-jailed), memory notes (as keyword/token-overlap notes — the importer warns that embedding-backed semantic recall does not carry over), cron jobs, timers (OpenClaw heartbeat → an interval cron), and tools (mcpServers/mcp_servers → MCP servers, imported needs_approval, never auto). Both parsers are defensive (OpenClaw JSON5, Hermes YAML + profiles/ dirs) and warn on anything they can't map. Cron specifics: an imported job keeps its source timezone (see below), and its last-run time is carried so it doesn't play catch-up on import; a schedule that can't become a 5-field cron (a one-shot, or an odd interval) is skipped with a warning, and a script/command job is imported disabled. --dry-run previews with zero writes; existing agents/cron/tools are kept unless --force. Because tool paths now expand a leading ~ to the leashed user's home at execution, imported skills that use ~/… paths work without rewriting.

Scheduled jobs carry an optional per-job IANA timezone (sentinel cron add --timezone, or preserved on import): the scheduler evaluates the cron expression in that zone via zoneinfo, so a job pinned to e.g. America/New_York fires at that wall-clock time wherever the host runs, with daylight-saving handled per instant. No timezone = the server's local zone (the prior default).

Operating it

The sentinel CLI: chat [--new], chats, restore <id>, setup, init, run "<task>" [--agent <name>], runs, show <id>, approvals, approve <id> / deny <id>, resume <id>, agent …, team …, workflow …, mcp …, model …, cron …, package export/import, import openclaw/import hermes, tools install-reqs, status (config + Muzzle/Leash health), preflight, and logs. Or drive all of it from the browser: sentinel web (control console) and sentinel api (programmable per-endpoint HTTP API). Both bind 127.0.0.1 by default and refuse a non-loopback bind unless you pass --allow-external — the console is plaintext HTTP, so exposing it is a deliberate choice to make behind TLS / a reverse proxy (or reach it over an SSH tunnel). The bearer token never appears in a URL: the SSE event stream and the VNC WebSocket authenticate with a single-use, short-lived stream ticket minted over the Authorization header, keeping the token out of logs and history.

Deployment profile + preflight. A profile config field declares the security level: production (and airgapped) require Muzzle (enabled + running), Leash (running), and Landlock (available, fs_sandbox on); development (the default) permits a degraded box but flags it. sentinel preflight checks the running host against the profile and exits non-zero when a required control is missing — turning the old silent downgrades (a muzzle-less box that still reported protected, a Leash that quietly no-oped, a kernel without Landlock that ran unsandboxed) into an explicit gate. The gate starts at install time: install.sh refuses to finish a production/airgapped box (set via SENTINEL_PROFILE, or inherited from an existing config) when Muzzle or Leash isn't a runnable install — so a degraded production deploy fails fast instead of surfacing later. Run preflight too, after install and in CI before a production cut-over. The web console surfaces the same assessment as a prominent banner (critical/warning/unknown) whenever the live posture is degraded for the profile, so it's visible without opening a terminal. Under production/airgapped this is also enforced at run start — a run whose required controls aren't active is recorded failed (with a security_preflight_failed event) instead of executing, and an insecure production config is rejected at load. Every confined run also emits a security_attestation event recording the posture it actually ran under — profile, leashed user + effective worker UID, the live Landlock result and ABI, the cgroup slice that bounds it, and Muzzle routing — so a run's own event stream is a verifiable audit record, not just the config it was launched with.