How Refinery works

Refinery fine-tunes models on compute you own, driven from the Sentinel console.

The shape of it

Sentinel host                          Compute target (yours)
─────────────                          ──────────────────────
corpus ──validate──scan──┐
                         └──ssh/scp──> /work/datasets/<name>.jsonl
job record (durable) ────ssh────────>  container: torch + peft/trl
   ▲                                      │
   └──reconcile (poll) ───────────────────┘ writes logs/<job>.log,
                                            artifacts/<job>/metrics.json

Also true of the transport line: v1 reaches a target through Sentinel's root broker by default, not by exec'ing ssh. Leash denies ssh to the leashed user deliberately — an agent that can ssh can pivot — so Refinery sends the broker a target NAME and the broker resolves the host, the port and the key from operator-owned config. SshTransport is the same Protocol for installs without a broker, and is selected by config.

Three stores live on the Sentinel host — targets, datasets and jobs — and a transport reaches the target. v1 ships two of them, broker and direct ssh, over one Protocol. That the second one already exists is the point: a token-authenticated cluster becomes a third implementation rather than an unpicking of everything that grew around ssh. The job store, the reconciler and the UI talk to the Protocol and have never seen an ssh command.

A job outlives the process that started it

This is the design constraint everything else follows from. A run is minutes; a training job is hours or days. The process that submits one is usually not the process that sees it finish.

So a job is a file, not an entry in memory. It records the target's own handle for the work — the container id — and that handle is what lets a restarted Sentinel find a job it never started. Opening the Refinery panel reconciles every live job against its target before painting, which is why a run that survived a sentinel-web restart reappears with its real state instead of frozen.

The job is written to disk before the target is contacted. A job that starts remotely with no local record is unreachable work: not cancellable, not reportable, still burning a GPU.

Absence is never read as success

The container runs with --rm, so a run that finishes cleanly and one that was OOM-killed leave the same empty machine behind. Completion is proven by the metrics.json the entrypoint writes last, atomically, only on success. Every failure path deliberately writes nothing.

A container that is gone with no metrics is reported as a failure with a legible reason, never as a success.

An unreachable target is not a verdict

A dropped VPN says nothing about the GPU. A failed poll leaves the job running rather than declaring a death that did not happen. A cancel that may not have landed says so, instead of showing a clean "cancelled" over a box that is still busy.

Datasets

A corpus is JSONL in one of three shapes — chat (messages), completion (prompt/completion), or plain text. All rows must use the same one: a mixed file trains on the subset the collator understands and silently drops the rest, which surfaces as a mediocre model rather than an error. Every rejection names the line.

Import validates, counts, scans for credentials, and writes a seeded train/val split. The validation set is not optional politeness: without a held-out set there is no signal separating a model that learned from one that memorised, so eval_loss is reported beside training loss and a tiny corpus gets one validation row rather than none.

The base model is a Hugging Face repo id, not an Ollama tag

The container calls AutoModelForCausalLM.from_pretrained(base_model), so the value must be something the Hub can resolve — Qwen/Qwen2.5-0.5B, not qwen2.5:0.5b. An Ollama tag names a GGUF in a local Ollama store, which is a serving format with no trainable weights in it. Refinery rejects tag-shaped values at submit rather than after a container start and a failed download.

The relationship runs the other way: Ollama is where a finished adapter goes, not where a base model comes from.

Chat templates: fill a gap, never overwrite

A messages corpus has to be rendered into text before it can be trained on, and that rendering is the tokenizer's chat template. Two cases, and the difference matters:

• The model ships its own template (instruct-tuned models, and some bases — Qwen2.5-0.5B carries a 2.4 KB one with tool-call support). Refinery leaves it completely alone. Overwriting it would train the adapter to speak a dialect its own base model does not, which produces a model that looks trained and answers badly.
• The model has none (gpt2, SmolLM-135M, most -base checkpoints). trl cannot render the corpus and raises an error pointing at its own docs, which tells an operator nothing about their corpus. Refinery supplies ChatML and says so loudly in the log, because the adapter then learns that format and the operator needs to know it when they serve it.

A completion or text corpus gets no template imposed at all — the decision reads the corpus, not the model.

The rendered template is saved into the artifact as chat_template.jinja, so what the adapter was trained to speak is recorded next to the weights rather than inferred later.

The dependency stack is resolved, not pinned

The image is pinned by digest, so the bytes are fixed. What is not fixed is the argument names inside those bytes: the NVIDIA base image ships its own transformers/trl/accelerate and bumps them on its own schedule, and those libraries rename constructor arguments between minor versions (max_seq_length→max_length, evaluation_strategy→eval_strategy, tokenizer→processing_class).

So the entrypoint asks the installed signature which name it takes instead of betting on a version. It also logs the resolved stack at the start of every run:

stack: torch=2.13.0a0+nv26.6 transformers=5.17.0 accelerate=1.15.0 peft=0.21.0 trl=1.13.0 datasets=5.0.0

A training log read weeks later then says what produced those weights, without anyone having to go and dissect an image.

A related trap worth naming: half-pinning is worse than not pinning. Pinning one library into an image whose others float produces a combination nobody tested, and it fails deep inside a call chain with a message about a keyword argument.

One log per job

Logs are per job — /work/logs/<job_id>.log — not a shared train.log. A single appended file interleaves every run that ever used the work dir, so jobs logs returned three jobs' output at once and a traceback could not be attributed to the run that produced it.

Outputs belong to the operator, not to root

The trainer runs as root inside the container, so everything it writes into the bind mount lands root-owned on the host. Left alone, the operator whose home holds the work dir cannot delete, move, quantise or serve the adapter they just trained.

So the entrypoint hands the artifact tree and the job log back to the uid that owns the work dir before it exits — on the failure path too, because a failed run's log is exactly the file the operator most needs to keep. It is best-effort and never fatal: a rootless or userns-remapped runtime neither needs it nor permits it.

Serving a finished adapter with Ollama

Refinery produces a LoRA adapter, which is not directly loadable by Ollama. The path from artifact to served model is three steps on the target:

1. Merge the adapter into the base model (peft's merge_and_unload).
2. Convert the merged model to GGUF (llama.cpp's convert_hf_to_gguf.py), quantising if you want it small.
3. ollama create from a Modelfile pointing at the GGUF.

Set the Modelfile's TEMPLATE to match what the adapter was trained on — that is what chat_template.jinja in the artifact is for. A model served with a different template than it was trained with is the single most common way a fine-tune that trained correctly answers badly.

Refinery does not automate these steps in v1.

What you get back

In v1, the weights stay on the target. A finished job records the artifact path, the metrics, and the digest of the image that produced them. That is a deliberate scope choice: adapters are megabytes but full fine-tunes are tens of gigabytes, and moving them back through the sandbox is a separate problem worth solving on its own.

Deleting weights

jobs purge-artifact <id> removes one job's weights from the target. Three decisions in it are worth stating, because this is Refinery's only destructive verb:

• The job record survives. Metrics, the training log, the image digest and who submitted it are the audit trail for work that already happened. Reclaiming disk is not a reason to destroy it.
• The path is rebuilt, not read back. It comes from the target's configured work dir and the job id held locally — never from the artifact_path the container reported, even though Refinery wrote that too. A record is not a place to take rm -rf targets from.
• "Gone" and "could not be removed" are different answers. rm -rf exits 0 on a missing path and swallows a permission failure on the contents, so the command verifies the path is actually gone afterwards. A silent failure here would report a still-full disk as freed.

A running job is refused: its weights are still being written, and removing them mid-run produces a job that reports success over an artifact path with nothing behind it.

Methods

LoRA and QLoRA adapters are the default and the tested path. method: full exists in the spec and the container branches on it, so full fine-tuning is a configuration value rather than a second code path — but it needs far more VRAM and has not been exercised end to end.

What has actually been run

Every claim above about behaviour on real hardware comes from this matrix, run on an NVIDIA GB10 (Blackwell, sm_121, aarch64) against the pinned arm64 image. Recorded because "the tests pass" did not catch a single one of the bugs these runs found — each lived in the gap between Refinery's code and a dependency's real behaviour, which is a gap no unit test can see into.

Two results are worth reading rather than skimming.

Loss tracks model capability — 10.7 for a randomly-initialised stub, 6.0 for SmolLM, 5.2 for Qwen. Training is doing real work rather than going through the motions, which a green "succeeded" alone would not tell you.

The last row is the only one where eval loss exceeds training loss (6.63 vs 6.36). That corpus is 38 rows of near-identical text, so the model memorised instead of learning — which is exactly what a held-out split exists to reveal, and the reason Refinery does not report training loss on its own.

Artifact inspection confirmed the template rules directly: the Qwen adapter carried Qwen's own 2427-byte template, the SmolLM adapters carried the injected 198-byte ChatML, and neither completion-corpus artifact contained a chat_template.jinja at all.