How Refinery works

Refinery fine-tunes models on compute you own, driven from the Sentinel console.

The shape of it

Sentinel host                          Compute target (yours)
─────────────                          ──────────────────────
corpus ──validate──scan──┐
                         └──ssh/scp──> /work/datasets/<name>.jsonl
job record (durable) ────ssh────────>  container: torch + peft/trl
   ▲                                      │
   └──reconcile (poll) ───────────────────┘ writes train.log + metrics.json

Three stores live on the Sentinel host — targets, datasets and jobs — and a transport reaches the target. v1 implements SSH; the transport is an interface so a token-authenticated cluster becomes a second implementation rather than a rewrite of everything built around ssh.

A job outlives the process that started it

This is the design constraint everything else follows from. A run is minutes; a training job is hours or days. The process that submits one is usually not the process that sees it finish.

So a job is a file, not an entry in memory. It records the target's own handle for the work — the container id — and that handle is what lets a restarted Sentinel find a job it never started. Opening the Refinery panel reconciles every live job against its target before painting, which is why a run that survived a sentinel-web restart reappears with its real state instead of frozen.

The job is written to disk before the target is contacted. A job that starts remotely with no local record is unreachable work: not cancellable, not reportable, still burning a GPU.

Absence is never read as success

The container runs with --rm, so a run that finishes cleanly and one that was OOM-killed leave the same empty machine behind. Completion is proven by the metrics.json the entrypoint writes last, atomically, only on success. Every failure path deliberately writes nothing.

A container that is gone with no metrics is reported as a failure with a legible reason, never as a success.

An unreachable target is not a verdict

A dropped VPN says nothing about the GPU. A failed poll leaves the job running rather than declaring a death that did not happen. A cancel that may not have landed says so, instead of showing a clean "cancelled" over a box that is still busy.

Datasets

A corpus is JSONL in one of three shapes — chat (messages), completion (prompt/completion), or plain text. All rows must use the same one: a mixed file trains on the subset the collator understands and silently drops the rest, which surfaces as a mediocre model rather than an error. Every rejection names the line.

Import validates, counts, scans for credentials, and writes a seeded train/val split. The validation set is not optional politeness: without a held-out set there is no signal separating a model that learned from one that memorised, so eval_loss is reported beside training loss and a tiny corpus gets one validation row rather than none.

What you get back

In v1, the weights stay on the target. A finished job records the artifact path, the metrics, and the digest of the image that produced them. That is a deliberate scope choice: adapters are megabytes but full fine-tunes are tens of gigabytes, and moving them back through the sandbox is a separate problem worth solving on its own.

Methods

LoRA and QLoRA adapters are the default and the tested path. method: full exists in the spec and the container branches on it, so full fine-tuning is a configuration value rather than a second code path — but it needs far more VRAM and has not been exercised end to end.