How Hurricane Sentinel Works

Summary

Sentinel is a local agent harness for Linux. A supervisor runs each agent as a Leash-leashed user with model calls pointed at a Muzzle listener, so enforcement is genuine: Leash gates the agent's file/network/command access per-UID and Muzzle inspects its model I/O. Runs are durable and checkpointed, and a flagged action pauses the run for a human approval before it proceeds.

The whole control plane (supervisor, web, API, cron) runs as the leashed user, not root — a worker is a same-user subprocess of it. The one operation that still needs root — changing Leash egress policy — is brokered by a tiny root sentinel-broker service over a root:<leashed> socket with two schema-validated verbs and no arbitrary execution; it is the only Sentinel service that runs as root.

The supervised run

1. Supervisor (root). The sentinel service owns run lifecycle and the approval queue. For each run it drops privileges and spawns a worker process as the leashed user, with the worker's model endpoint set to a Muzzle listener.
2. Run engine (durable + checkpointed). A run is one agent task. Its state — message history, step, status — is persisted to the leashed user's ~/.sentinel/runs/<id>/. Status moves running → waiting_approval → done | failed, and every step (and every pause) is checkpointed so a run resumes exactly where it stopped.
3. Agent worker (as the leashed user). The loop: build messages → call the model through Muzzle → read the model's tool calls → for each, consult the action policy:
   • auto → run the tool (its file/network/command effects are enforced by Leash);
   • needs_approval → checkpoint, enqueue an approval, and pause the run;
   • deny → feed the denial back to the agent. No tool calls → the model's answer is final → the run is done.
4. Async approvals. Pending approvals are persisted. An operator lists them and approves or denies via the CLI; on approval the supervisor resumes the run and the tool executes, then the loop continues.

When a guardrail blocks

Enforcement is only half the story — a block the agent can't understand just makes it fail or loop. So every block is made legible to both the agent and the operator, without adding any new execution path:

• Muzzle blocks (model I/O). A rejected model call is typed by direction and category (secrets / PII / prompt-injection / content-policy). If the block is on the agent's output, the worker feeds it a rewrite note and re-drives the turn (bounded by a per-run retry cap) so it can produce a compliant reply; exhausting the cap fails the run cleanly. If the block is on the input it was handed, the agent can't process it — the run stops with a plain explanation and does not retry.
• Leash / sandbox denials (OS actions). A tool failure is classified: a filesystem denial (Landlock or Leash's fs surface) comes back to the model as the path is outside your workspace — write to your workspace or /tmp, and a blocked network reach comes back as the host isn't on the operator's allow-list — legible guidance, not a raw errno.
• Operator visibility. Each block emits a guardrail_block event that the web console renders as a color-coded card in the run feed (source-tinted for Muzzle vs Leash/sandbox). For a Leash denial the supervisor (leashed) correlates the event with Leash's decision log and attaches the exact reason (e.g. "1.1.1.1:80 not in whitelist") — the correlation is a supervisor concern, done once outside the run, not per worker.
• In-context guidance. When a run is subject to Muzzle and/or the sandbox, a compact guardrails skill is auto-injected into the agent's system prompt (only the parts that apply to that run), teaching it what each block means and how to respond — rewrite, stop-and-explain, or pick an allowed path/host.

This is pure observability layered on top of the existing enforcement: a guardrail_block never executes the blocked action, the injected skill is text only (it grants no tools), and the correlation with Leash's log happens root-side only. Making a block visible never makes it bypassable.

Why files live in the leashed user's home

Per-agent runtime state (souls/, skills/, memory/, runs/) lives under the leashed user's ~/.sentinel, so the worker — which runs as that user under Leash's filesystem enforcement — can read and write its own state without anyone having to open up a privileged system directory.

Dependencies

Sentinel requires Muzzle and Leash. The installer verifies (and can install) both, and sentinel init --leash <user> [--muzzle] registers the agent's user with Leash, ensures Leash is running, and checks Muzzle.

Identity: one name for the agent and its user

First-run setup asks for one name (sentinel setup prompts "Name your agent (also its Linux user)"; headless installs default to naomi). That single name becomes both the head agent and the leashed Linux user: the display name is preserved for the agent (e.g. Naomi), and a normalized form (lowercase, [a-z0-9_]) is the OS account (e.g. naomi). There is no generic main agent anymore — init creates the named head agent, points default_agent at it, and writes its starter soul.md. --leash <name> doubles as the name on either command.

Migration: existing deployments keep their current leashed user and main agent until you re-run setup with a name (which creates the new named head agent alongside main — nothing is renamed or deleted).

Self-state: soul, operator model, and memory

The head agent maintains its own context across sessions through three self-state files under ~/.sentinel/, all loaded into the system prompt each session:

• agents/<name>.soul.md — its persona (overrides the profile soul). Edited via the edit_soul tool (auto; each write is backed up to .soul.md.bak).
• user.md — its model of the operator, shared across the head agent. Appended via the update_user tool (auto).
• memory — durable notes via remember / recall (see below).

All four self-state/memory tools run auto (no per-edit approval) so the agent can keep itself current without interrupting the operator.

Agents author their own skills and tools

An agent can extend itself at runtime, not just remember things:

• Skills are markdown. create_skill writes ~/.sentinel/skills/<name>/SKILL.md (frontmatter + body) and attaches it to the authoring agent; it loads into that agent's system prompt on the next turn. Skills carry no execution, so create_skill / list_skills run auto.
• Tools are Python. create_tool writes ~/.sentinel/tools/<name>/{tool.yaml,tool.py} (the code must define run(args) returning a string). The tool then loads into the registry like any built-in, and a call runs the code out-of-process as the leashed user — so it inherits the same Leash egress allowlist and Landlock filesystem sandbox as the worker, plus a CPU-time limit and a wall-clock timeout. The built-in run_command tool runs through the same bounded runner the whole harness is standardizing on: a wall-clock timeout, a whole-process-group kill so a command that forks children can't survive the timeout, a CPU ceiling, and an in-memory output cap. (Hard RSS/memory bounding is a deployment control — a systemd/cgroup MemoryMax; see SECURITY.md.) A failure never crashes the run: a sandbox/Leash denial comes back as a legible guardrail reason (and a guardrail_block event — see When a guardrail blocks), while other failures (timeout, exception) come back as an error: … string fed to the model. create_tool / edit_tool / list_tools / delete_tool run auto — managing a tool is not the same as running it.

Approve-once-then-auto. Running an authored tool is gated by trust keyed on a SHA-256 of its code. The first call to a freshly authored (or freshly edited) tool pauses for approval, showing the operator the code they're about to trust; approving runs it and marks that exact code trusted, so later calls run automatically. Editing the tool clears trust (the code changed), so the next call re-prompts with the new code. Denial does not trust it. This keeps the "agent evolves itself" loop fast while keeping a human in the loop the first time any new code would run.

Persistent agents and orchestration

An agent is a named, on-disk profile under the leashed user's ~/.sentinel/agents/: a soul (persona / system prompt), a bound model chosen from your configured list (routed via Muzzle), an extensible skill set (SKILL.md files injected into its context), and the tools it may call. Run one with sentinel run --agent <name>.

One agent acts as the orchestrator. Given a task it can:

• delegate the task to a persistent specialist it picks from the roster, or
• spawn a temporary subagent — composing a persona, model, and skills on the fly — when no specialist fits.

A delegated or spawned subagent runs as its own durable child run under the same Muzzle

• Leash enforcement (possibly a different model), and returns its result to the parent. Delegation is the approval gate (you approve each delegation, seeing the agent + task + grants); within it the child's granted tools run automatically. Tool grants flow parent → child (least privilege), depth is capped, and a temporary subagent is discarded, saved to the roster, or queued for your approval (ask) per config.

Reliable on small models

Because small local models are inconsistent at tool calling, the harness compensates: a tool-use prompt scaffold tells the model to emit real tool calls; the gateway recovers calls a model emits as plain text; tool arguments are normalized and validated against their schema, with a clear correction fed back so the model retries; and a subagent automatically receives the orchestrator's original request so a terse hand-off never loses detail.

Memory and context compaction

Each agent has its own memory bank under ~/.sentinel/memory/<scope>/ — a flat index.md plus an Obsidian-style vault/ of notes (YAML frontmatter + body + [[links]]). The scope comes from the agent's profile (memory: field): private to the agent by default, or shared when several agents name the same scope. The agent uses two tools: remember(title, content, tags?, links?) writes a bounded note, and recall(query) performs model-agnostic lexical retrieval with stopword filtering, light inflection normalization, title/tag/phrase weighting, and recency tie-breaking. Relevant memories are also surfaced at the start of a task, but are explicitly marked as untrusted and potentially stale context. Knowledge persists across runs without requiring an embedding service. Semantic/embedding recall remains a later enhancement.

Context compaction keeps long runs inside the model's window: when a run's history crosses a (character-estimated, tokenizer-agnostic) threshold, Sentinel summarizes the older middle turns into one running "summary so far" message via a model call, keeps the system prompt (soul + skills) and the most recent turns verbatim, and checkpoints the result.

Validated today

Built and validated on Linux end-to-end: supervised runs (as the leashed user, model via Muzzle) that pause on a needs_approval action and resume on approval — including with Leash filesystem enforcement live; a main agent delegating to a persistent specialist that runs as a parent-linked child and produces the correct result on a small local model; and an agent remembering a fact in one run and recalling it correctly in a later one. A local web UI is the remaining milestone.

The interactive shell

sentinel chat opens a colorized REPL for talking to an agent. Run it as the leashed user (sudo -u <user> sentinel chat) so the agent's tools execute under that account and write to its workspace. The shell:

• streams live progress (each model call and tool call) and types out the final, Muzzle-cleared answer;
• handles approvals inline — a flagged action prompts y/N right there (the async queue is still there for headless run);
• offers completion menus as you type: / for commands (/help, /agents, /use, /model, /status, /memory, /skills, /new, /quit…), @name to send a task to a specific specialist, and $ to browse and attach skills;
• keeps ↑/↓ history across sessions and colorizes commands as you type.

The same loop that powers headless runs drives the chat, so durable state, memory, delegation, and the small-model safeguards (tool-call dedup, the bundled file-discipline skill) all apply. Agents work from a per-user workspace, so a bare filename like notes.txt lands somewhere the leashed user owns.

Chat sessions are durable and resume by default. A plain sentinel chat auto-resumes the most recent session (with a one-line banner); sentinel chat --new (or /new in the shell) starts fresh. sentinel chats lists saved sessions by id and title, and sentinel restore <id> reopens a specific one. Each session is a durable run, so its history and self-state survive across invocations.

Saved workflows

A workflow is a saved, named pipeline at ~/.sentinel/workflows/<name>.yaml. Each step names an agent and a task; the runner executes the steps in order, and outputs thread into later steps through templating:

• {{input}} — the value passed to workflow run --input,
• {{<step>.output}} — a named earlier step's result,
• {{previous}} — the immediately prior step's result.

Each step runs as its own supervised run with its agent (so Muzzle inspection and Leash enforcement apply, and the step's granted tools run within the workflow's authorization). Trigger a workflow with sentinel workflow run <name> [--input …], the chat's /workflow run <name>, or let an agent invoke the run_workflow(name, input) tool as part of its own work (gated by approval policy, like delegate). Workflows are linear in v1 — no branching or loops yet.

Teams

A team is a leader-led roster with a shared workspace, an authored guidelines.md (prepended to every team-scope run), and an activity log — persisted under ~/.sentinel/teams/<name>/. The head agent delegates a task to a team; that spawns the team's leader as a team:<name>-scoped child run, and the leader in turn delegates to its members (delegate_to_member(s)) — each a scoped, sandboxed, approval-gated child run of its own. The supervisor drives the members concurrently in a bounded pool, joins their results back to the leader, and the leader synthesizes an answer. The boundaries are hard: a member's Landlock sandbox excludes other teams' and other agents' private workspaces, cross-team delegation is only reachable from the head, fan-out is bounded (concurrency cap + a per-round item cap), and a hard-killed child is reaped so the leader never hangs waiting on it.

Model providers (local + external)

Every model call goes through a Muzzle listener. Local Ollama is the default; an external provider (OpenAI today) is added as a keyless model alias whose per-provider listener + API key Sentinel pushes into Muzzle — keys live only in Muzzle, never in Sentinel's model store or its API responses. The worker picks the right gateway from the model's resolved provider and dials the listener on loopback. A Muzzle-less install can point straight at Ollama (no external providers, no scrubbing) for a minimal box.

Uploaded tools (the MCP tool host)

Beyond built-in and agent-authored tools, an operator can upload a plain .py file. The root process AST-scans it (never importing it) and lists its functions; the operator picks which become tools, and Sentinel appends a managed marker to those functions. A private, built-in stdio MCP server (sentinel-uploads, seeded automatically) serves the marked functions to agents as mcp__sentinel-uploads__<file>__<func> — approval-gated like any MCP tool. Each call runs in a short-lived, killable subprocess (the leashed host spawns one per call and re-checks the managed marker there before importing), so a timeout terminates the work — a runaway or forking uploaded tool can't keep running in the host after its deadline. The file's third-party requirements are detected and installed on one click (leashed pip into a per-user site dir). These tools travel with portability exports.

Structured endpoints (validated input + output)

A programmable endpoint (sentinel api, POST /api/<name> with a per-endpoint bearer key) turns a request into a normal supervised run — and can shape both ends with an uploaded Pydantic class. The operator uploads a plain .py of BaseModel classes in the web UI; the root process never imports it — a leashed subprocess introspects it (its classes and each class's fields, declared + computed) and records them.

• Structured output: pick an output class for the endpoint. When the agent's run finishes, a leashed with_structured_output(<class>) call — using the agent's own provider, routed through the Muzzle listener like every model call — coerces the answer into a validated instance and returns it as JSON. A model that can't be forced (or a missing class) yields a clean 502, never a crash.
• Structured input: pick an input class. Each call validates the request input against it in a leashed subprocess; a Pydantic error returns a 400 naming the field, and no run is created. The endpoint's single prompt template then lays the validated data out for the model with %field% tokens — scalars inline, lists/objects as JSON, and computed fields (e.g. a transcript derived from a messages list) for custom formatting. The web UI offers each class's fields as insertable chips and highlights valid vs. unknown %tokens% as you type.

Untrusted uploaded classes execute only in the leashed sandbox — at upload (introspect), and per request (validate + coerce) — never in the root web/api process. Endpoints without a selected structure keep their prior free-text behavior.

Portability

Export an agent or a whole team as a secret-free .sentinelpkg (a zip): its skills, authored tools, MCP configs with secrets scrubbed, keyless model aliases, souls, team definition + guidelines, and any uploaded tool files + a recomputed requirements list. Import it into another deployment — collisions resolve by operator-chosen rename with all cross-references rewritten, imported tools arrive untrusted and MCP servers disabled, the reserved uploads host is never overwritten, and fresh workspaces are scaffolded. The receiver supplies their own keys and installs requirements, and the team runs.

Importing from OpenClaw and Hermes

sentinel import openclaw <path> / sentinel import hermes <path> (also More ▾ → Import agent in the console, which takes a zipped home) migrate an existing assistant into native Sentinel objects: agents (model resolved against config.models), skills (SKILL.md + resources, source-only frontmatter stripped, resources path-jailed), memory notes (as keyword/token-overlap notes — the importer warns that embedding-backed semantic recall does not carry over), cron jobs, timers (OpenClaw heartbeat → an interval cron), and tools (mcpServers/mcp_servers → MCP servers, imported needs_approval, never auto). Both parsers are defensive (OpenClaw JSON5, Hermes YAML + profiles/ dirs) and warn on anything they can't map. Cron specifics: an imported job keeps its source timezone (see below), and its last-run time is carried so it doesn't play catch-up on import; a schedule that can't become a 5-field cron (a one-shot, or an odd interval) is skipped with a warning, and a script/command job is imported disabled. --dry-run previews with zero writes; existing agents/cron/tools are kept unless --force. Because tool paths now expand a leading ~ to the leashed user's home at execution, imported skills that use ~/… paths work without rewriting.

Scheduled jobs carry an optional per-job IANA timezone (sentinel cron add --timezone, or preserved on import): the scheduler evaluates the cron expression in that zone via zoneinfo, so a job pinned to e.g. America/New_York fires at that wall-clock time wherever the host runs, with daylight-saving handled per instant. No timezone = the server's local zone (the prior default).

Operating it

The sentinel CLI: chat [--new], chats, restore <id>, setup, init, run "<task>" [--agent <name>], runs, show <id>, approvals, approve <id> / deny <id>, resume <id>, agent …, team …, workflow …, mcp …, model …, cron …, package export/import, import openclaw/import hermes, tools install-reqs, status (config + Muzzle/Leash health), preflight, and logs. Or drive all of it from the browser: sentinel web (control console) and sentinel api (programmable per-endpoint HTTP API). Both bind 127.0.0.1 by default and refuse a non-loopback bind unless you pass --allow-external — the console is plaintext HTTP, so exposing it is a deliberate choice to make behind TLS / a reverse proxy (or reach it over an SSH tunnel). The bearer token never appears in a URL: the SSE event stream and the VNC WebSocket authenticate with a single-use, short-lived stream ticket minted over the Authorization header, keeping the token out of logs and history.

Deployment profile + preflight. A profile config field declares the security level: production (and airgapped) require Muzzle (enabled + running), Leash (running), and Landlock (available, fs_sandbox on); development (the default) permits a degraded box but flags it. sentinel preflight checks the running host against the profile and exits non-zero when a required control is missing — turning the old silent downgrades (a muzzle-less box that still reported protected, a Leash that quietly no-oped, a kernel without Landlock that ran unsandboxed) into an explicit gate. The gate starts at install time: install.sh refuses to finish a production/airgapped box (set via SENTINEL_PROFILE, or inherited from an existing config) when Muzzle or Leash isn't a runnable install — so a degraded production deploy fails fast instead of surfacing later. Run preflight too, after install and in CI before a production cut-over. The web console surfaces the same assessment as a prominent banner (critical/warning/unknown) whenever the live posture is degraded for the profile, so it's visible without opening a terminal. Under production/airgapped this is also enforced at run start — a run whose required controls aren't active is recorded failed (with a security_preflight_failed event) instead of executing, and an insecure production config is rejected at load. Every confined run also emits a security_attestation event recording the posture it actually ran under — profile, leashed user + effective worker UID, the live Landlock result and ABI, the cgroup slice that bounds it, and Muzzle routing — so a run's own event stream is a verifiable audit record, not just the config it was launched with.