How Learner Works

Learner is a pipeline from sources to a knowledge base to a reusable skill. You feed it material on a topic; it stores a searchable, distilled memory of that material and can synthesize the memory into a skill an agent uses.

The ingest pipeline

learner ingest --topic <t> --resource <value> --kind <kind> runs each source through these stages:

1. Load. Resolve the source by kind:
   • article_url — fetch and extract the article text.
   • video_url — pull a YouTube transcript.
   • file_path — read a local file (plain text, or a PDF via pypdf).
   • text — take the value inline.
2. Refine (optional). With --refine, an LLM rewrites the source into a high-signal retrieval document — dropping ads, boilerplate, and off-topic material while preserving constraints, definitions, and procedures. The refined text is what gets indexed.
3. Chunk. The (refined or raw) text is split into overlapping chunks.
4. Embed. Chunks are embedded with the configured embeddings backend.
5. Store. Chunks + embeddings go into a per-topic SQLite knowledge base under data_dir (default ~/.learner/knowledge_base.sqlite3), keyed by a stable source id and a checksum so re-ingesting an unchanged source is a no-op.
6. Distill knowledge items. Unless --no-knowledge is set, each chunk is also passed to an LLM that extracts durable knowledge items — typed as rule, workflow, definition, warning, example, or note, each with a confidence and an inference flag. These are embedded and stored too.

ingest accepts one --resource at a time (repeatable) or a JSON --manifest ({topic, resources:[{title,kind,value,...}]}). refresh --topic <t> re-runs the whole pipeline over every source already stored for a topic; list shows them; export writes a JSON migration bundle of a topic's vectors and sources.

Retrieval (RAG)

Two commands read the KB:

• query --question <q> [--topic <t>] — embeds the question and returns the top semantic matches (chunks and knowledge items), each with a score.
• rag-answer --question <q> [--topic <t>] — retrieves context, then asks the chat model to answer using only that context, ending with a Sources line that cites the retrieved items by bracket number.

Building a skill

build-skill --topic <t> [--out <dir>] [--zip] synthesizes the topic's stored knowledge into a playbook:

1. A TopicProfile is generated for the topic (LLM, with a deterministic domain heuristic fallback for software / business / health / general) — it decides the playbook title and its section layout.
2. The stored knowledge items are grouped by kind into a takeaways block and fed to the synthesizer, which merges duplicates, resolves conflicts, and writes a single dense SKILL.md — profile-driven sections plus a block of explicit if-then rules the agent can act on.
3. The result is written as <name>.skill.md (or <name>.skill.zip with a top-level SKILL.md) with YAML frontmatter (name, description).

The frontmatter and archive layout match Sentinel's skill importer exactly, so a learned skill imports into Sentinel unchanged.

Configuration

Config is a small YAML file (~/.learner/learner.yaml or /etc/learner/learner.yaml; learner init seeds one and creates the data dir). Sections:

• data_dir — where the SQLite KB lives (default ~/.learner).
• llm — endpoint is direct (talk to base_url below) or muzzle://<host:port> (route model calls through a Muzzle listener so they're inspected and keys stay in Muzzle); chat_provider is ollama or openai; plus chat_model and base_url.
• embeddings — backend is ollama (default), hf, or openai, plus model and an optional base_url. The base install carries only the Ollama path; hf and openai are opt-in extras (learner[hf] / learner[openai]) so an install stays slim unless you need them.

Install

install.sh is a CLI install only — no systemd service. It lays down /opt/learner/v1 and a virtualenv, puts a learner command on PATH (/usr/local/bin/learner), and seeds /etc/learner/learner.yaml (preserving an existing config on reinstall). LEARNER_INSTALL_EXTRAS=hf (or openai) installs the matching embeddings extra. Requires python3.12+.