How Refinery works

Refinery fine-tunes models on compute you own, driven from the Sentinel console.

The shape of it

Sentinel host                          Compute target (yours)
─────────────                          ──────────────────────
corpus ──validate──scan──┐
                         └──ssh/scp──> /work/datasets/<name>.jsonl
job record (durable) ────ssh────────>  container: torch + peft/trl
   ▲                                      │
   └──reconcile (poll) ───────────────────┘ writes logs/<job>.log,
                                            artifacts/<job>/metrics.json

Also true of the transport line: v1 reaches a target through Sentinel's root broker by default, not by exec'ing ssh. Leash denies ssh to the leashed user deliberately — an agent that can ssh can pivot — so Refinery sends the broker a target NAME and the broker resolves the host, the port and the key from operator-owned config. SshTransport is the same Protocol for installs without a broker, and is selected by config.

Three stores live on the Sentinel host — targets, datasets and jobs — and a transport reaches the target. v1 ships two of them, broker and direct ssh, over one Protocol. That the second one already exists is the point: a token-authenticated cluster becomes a third implementation rather than an unpicking of everything that grew around ssh. The job store, the reconciler and the UI talk to the Protocol and have never seen an ssh command.

A job outlives the process that started it

This is the design constraint everything else follows from. A run is minutes; a training job is hours or days. The process that submits one is usually not the process that sees it finish.

So a job is a file, not an entry in memory. It records the target's own handle for the work — the container id — and that handle is what lets a restarted Sentinel find a job it never started. Opening the Refinery panel reconciles every live job against its target before painting, which is why a run that survived a sentinel-web restart reappears with its real state instead of frozen.

The job is written to disk before the target is contacted. A job that starts remotely with no local record is unreachable work: not cancellable, not reportable, still burning a GPU.

Absence is never read as success

The container runs with --rm, so a run that finishes cleanly and one that was OOM-killed leave the same empty machine behind. Completion is proven by the metrics.json the entrypoint writes last, atomically, only on success. Every failure path deliberately writes nothing.

A container that is gone with no metrics is reported as a failure with a legible reason, never as a success.

An unreachable target is not a verdict

A dropped VPN says nothing about the GPU. A failed poll leaves the job running rather than declaring a death that did not happen. A cancel that may not have landed says so, instead of showing a clean "cancelled" over a box that is still busy.

Datasets

A corpus is JSONL in one of three shapes — chat (messages), completion (prompt/completion), or plain text. All rows must use the same one: a mixed file trains on the subset the collator understands and silently drops the rest, which surfaces as a mediocre model rather than an error. Every rejection names the line.

A corpus can be dropped onto the Datasets tab rather than placed on the host and named by path. It streams to disk — a training corpus runs to hundreds of megabytes and the control plane should never hold one in memory — then is validated and scanned before anything is imported. Staging is not importing: the file lands in the leashed user's own state dir, and it becomes a dataset only when an approver says so, which is where the export gate still sits. A failed import keeps the upload, so fixing a name or ticking the override does not mean sending it again.

Import validates, counts, scans for credentials, and writes a seeded train/val split. The validation set is not optional politeness: without a held-out set there is no signal separating a model that learned from one that memorised, so eval_loss is reported beside training loss and a tiny corpus gets one validation row rather than none.

The base model is a Hugging Face repo id, not an Ollama tag

The container calls AutoModelForCausalLM.from_pretrained(base_model), so the value must be something the Hub can resolve — Qwen/Qwen2.5-0.5B, not qwen2.5:0.5b. An Ollama tag names a GGUF in a local Ollama store, which is a serving format with no trainable weights in it. Refinery rejects tag-shaped values at submit rather than after a container start and a failed download.

The relationship runs the other way: Ollama is where a finished adapter goes, not where a base model comes from.

Chat templates: fill a gap, never overwrite

A messages corpus has to be rendered into text before it can be trained on, and that rendering is the tokenizer's chat template. Two cases, and the difference matters:

• The model ships its own template (instruct-tuned models, and some bases — Qwen2.5-0.5B carries a 2.4 KB one with tool-call support). Refinery leaves it completely alone. Overwriting it would train the adapter to speak a dialect its own base model does not, which produces a model that looks trained and answers badly.
• The model has none (gpt2, SmolLM-135M, most -base checkpoints). trl cannot render the corpus and raises an error pointing at its own docs, which tells an operator nothing about their corpus. Refinery supplies ChatML and says so loudly in the log, because the adapter then learns that format and the operator needs to know it when they serve it.

A completion or text corpus gets no template imposed at all — the decision reads the corpus, not the model.

The rendered template is saved into the artifact as chat_template.jinja, so what the adapter was trained to speak is recorded next to the weights rather than inferred later.

The dependency stack is resolved, not pinned

The image is pinned by digest, so the bytes are fixed. What is not fixed is the argument names inside those bytes: the NVIDIA base image ships its own transformers/trl/accelerate and bumps them on its own schedule, and those libraries rename constructor arguments between minor versions (max_seq_length→max_length, evaluation_strategy→eval_strategy, tokenizer→processing_class).

So the entrypoint asks the installed signature which name it takes instead of betting on a version. It also logs the resolved stack at the start of every run:

stack: torch=2.13.0a0+nv26.6 transformers=5.17.0 accelerate=1.15.0 peft=0.21.0 trl=1.13.0 datasets=5.0.0

A training log read weeks later then says what produced those weights, without anyone having to go and dissect an image.

A related trap worth naming: half-pinning is worse than not pinning. Pinning one library into an image whose others float produces a combination nobody tested, and it fails deep inside a call chain with a message about a keyword argument.

The training image is pinned, and pinning has a cost

The image carries the training code — the entrypoint and the torch/peft/trl stack it runs against. It is pinned by DIGEST, never a tag, because a tag can be repointed and then nothing records what produced a set of weights. Each job records the digest it ran.

The cost is that it never moves on its own. Updating the add-on brings new training code and the digests that go with it, but the pin is operator configuration and stays where it was. Twice that left a working install running an image two fixes behind, with the failure naming a Python library rather than the image that was actually old.

So the lifecycle is explicit:

• Currency. refinery images compares the pin against the digests this build ships with, reported per architecture: current, stale, unset, unpinned (a tag — not out of date, unpinnable), or unknown for an operator's own build, which is never reported as stale. The expected digests come from the refinery.example.yaml inside the package, so the check needs no network.
• Repin. The Targets tab offers it against the target whose architecture selects the image. Read-modify-write happens server-side, because the config holds keys the console never renders and a round trip through a form drops them.
• Fetch. Repinning changes a string; it does not put several gigabytes on a GPU box. Fetching is offered as a separate step so a first job does not spend twenty minutes pulling.
• Prune. Older images are full CUDA stacks and nothing else removes them. The listing is a separate verb from the removal — deleting gigabytes off someone else's machine is an answer to a question, never a side effect of looking. The pinned image is never a candidate, only refinery-train-* repositories are considered, and an image a running job still uses is reported as kept, not forced.

Two things learned the hard way here. Images are enumerated with image ls -a --digests: docker's --filter reference= matches repo:tag, and every image Refinery pulls is pulled by digest and therefore untagged. And removal takes every reference — an image rm repo@digest on an image that also has a tag drops only the digest, leaving the layers behind under a name nothing will offer again.

Sizes are virtual. Training images share nearly all their layers, so five images each reporting 23.9 GB are nowhere near 120 GB of reclaimable space. docker system df on the target is the honest number.

When peft has never heard of the model

LoRA needs to know which modules to adapt, and peft picks them from a table keyed on the model's architecture. A model newer than the installed peft is not in that table, and the run dies with Please specify target_modules — which names nothing about the model, the architecture, or what to do.

That is not an edge case: transformers learns an architecture long before peft's table does, so the newest models are precisely the ones that fail.

Refinery reads the layer names off the model instead. It is reactive — peft's own table stays authoritative for what it knows, and only the failure triggers the fallback — and it logs exactly which modules it chose, because that decides what the adapter is. lm_head is excluded: vocab-sized, not what LoRA normally means, and it would quietly multiply the adapter.

One log per job

Logs are per job — /work/logs/<job_id>.log — not a shared train.log. A single appended file interleaves every run that ever used the work dir, so jobs logs returned three jobs' output at once and a traceback could not be attributed to the run that produced it.

Outputs belong to the operator, not to root

The trainer runs as root inside the container, so everything it writes into the bind mount lands root-owned on the host. Left alone, the operator whose home holds the work dir cannot delete, move, quantise or serve the adapter they just trained.

So the entrypoint hands the artifact tree and the job log back to the uid that owns the work dir before it exits — on the failure path too, because a failed run's log is exactly the file the operator most needs to keep. It is best-effort and never fatal: a rootless or userns-remapped runtime neither needs it nor permits it.

Serving a finished adapter with Ollama

Refinery produces a LoRA adapter, which is not directly loadable by Ollama. The path from artifact to served model is three steps on the target:

1. Merge the adapter into the base model (peft's merge_and_unload).
2. Convert the merged model to GGUF (llama.cpp's convert_hf_to_gguf.py), quantising if you want it small.
3. ollama create from a Modelfile pointing at the GGUF.

Set the Modelfile's TEMPLATE to match what the adapter was trained on — that is what chat_template.jinja in the artifact is for. A model served with a different template than it was trained with is the single most common way a fine-tune that trained correctly answers badly.

Refinery does not automate these steps in v1.

What you get back

In v1, the weights stay on the target. A finished job records the artifact path, the metrics, and the digest of the image that produced them. That is a deliberate scope choice: adapters are megabytes but full fine-tunes are tens of gigabytes, and moving them back through the sandbox is a separate problem worth solving on its own.

Deleting weights

jobs purge-artifact <id> removes one job's weights from the target. Three decisions in it are worth stating, because this is Refinery's only destructive verb:

• The job record survives. Metrics, the training log, the image digest and who submitted it are the audit trail for work that already happened. Reclaiming disk is not a reason to destroy it.
• The path is rebuilt, not read back. It comes from the target's configured work dir and the job id held locally — never from the artifact_path the container reported, even though Refinery wrote that too. A record is not a place to take rm -rf targets from.
• "Gone" and "could not be removed" are different answers. rm -rf exits 0 on a missing path and swallows a permission failure on the contents, so the command verifies the path is actually gone afterwards. A silent failure here would report a still-full disk as freed.

A running job is refused: its weights are still being written, and removing them mid-run produces a job that reports success over an artifact path with nothing behind it.

Methods

LoRA and QLoRA adapters are the default and the tested path. method: full exists in the spec and the container branches on it, so full fine-tuning is a configuration value rather than a second code path — but it needs far more VRAM and has not been exercised end to end.

What has actually been run

Every claim above about behaviour on real hardware comes from this matrix, run on an NVIDIA GB10 (Blackwell, sm_121, aarch64) against the pinned arm64 image. Recorded because "the tests pass" did not catch a single one of the bugs these runs found — each lived in the gap between Refinery's code and a dependency's real behaviour, which is a gap no unit test can see into.

Corpus	Shape	Base model	loss → eval	What it proves
114 rows	chat	sshleifer/tiny-gpt2	10.73 → 10.73	the contract, end to end
76 rows	completion	sshleifer/tiny-gpt2	10.82 → 10.83	no template imposed
114 rows	chat	HuggingFaceTB/SmolLM-135M	6.03 → 5.95	real weights; ChatML injected
114 rows	chat	Qwen/Qwen2.5-0.5B	5.24 → 4.75	model's own template preserved
76 rows	completion	HuggingFaceTB/SmolLM-135M	5.64 → 5.51	shape × model cross-check
38 rows	chat, scanner-tripping	HuggingFaceTB/SmolLM-135M	6.36 → 6.63	the override path

Two results are worth reading rather than skimming.

Loss tracks model capability — 10.7 for a randomly-initialised stub, 6.0 for SmolLM, 5.2 for Qwen. Training is doing real work rather than going through the motions, which a green "succeeded" alone would not tell you.

The last row is the only one where eval loss exceeds training loss (6.63 vs 6.36). That corpus is 38 rows of near-identical text, so the model memorised instead of learning — which is exactly what a held-out split exists to reveal, and the reason Refinery does not report training loss on its own.

Artifact inspection confirmed the template rules directly: the Qwen adapter carried Qwen's own 2427-byte template, the SmolLM adapters carried the injected 198-byte ChatML, and neither completion-corpus artifact contained a chat_template.jinja at all.