Fine-tuning Jupiter-N on eight DGX Sparks
What happens when you try to teach cynghanedd to a 120-billion-parameter model on eight desktop computers? An honest report on the work so far.
Bangor AI's experiments with the improvement cycle suggest that fine-tuning can improve the form of cynghanedd, but does not necessarily add the vocabulary or meaning missing from the original model. One approach to test is starting with a model already post-trained on Welsh. Jupiter-N-120B was chosen to investigate this approach at 120B scale.
The hardware: plenty of memory, limited bandwidth
Bangor AI's cluster consists of eight DGX Spark (GB10) units. Together they have around a terabyte of unified memory and 200GbE links between nodes, but each has only around 273 GB/s of memory bandwidth—roughly a quarter of a good gaming card's. In practice, the cluster has plenty of memory but modest compute. That can suit an MoE model such as Jupiter: all 120 billion parameters can remain in memory, while only the roughly 12 billion active parameters need computation. In an initial test, NCCL all-reduce across the network reached 93% of theoretical link speed, from two nodes up to eight.
Experiment 1 — LoRA distributed across eight nodes
A node with 121GB of memory cannot hold 240GB of bf16 weights, so weights were streamed tensor by tensor: one node read the data and distributed the appropriate shard to each other node. A custom LoRA adapter (r=16, α=32) was used at 112 sites in attention layers and hidden MoE projections. Training used cynghanedd-specific data, with evaluation every 100 steps: the model was asked for a seven-syllable line, judged by the rules engine .
The result so far: training reached step 245 of 363 before infrastructure failure. It scored 0/5 on valid-at-7 at every evaluation. The lines contained many cynghanedd faults, though some came very close: ‘Nadoli'g, nadolig’ was valid traws, but contained only six syllables. More revealingly, although a checking tool was available, the model never used the tool. It discussed using the tool in its reasoning, but never made the call. Fixing tool-call formatting—rather than poetic ability—is therefore a priority before restarting.
Experiment 2 — training on the 4-bit weights themselves
The second route is to compress Jupiter to NVFP4—around 63GB, small enough for one DGX Spark—and train LoRA directly on the 4-bit weights, including all 20,520 expert projections, without converting back to bf16. The tooling has already been tested on other model families. On Jupiter, the first training attempt ended at step 30 of 120, before the first checkpoint. Throughput of one or two training tokens per second suggested further investigation of batch size, checkpointing and feasibility was needed.
What next
- Fix tool calls before spending more cluster time—the cheapest defect to fix and potentially the most consequential.
- Restart cluster-wide training from the step-200 checkpoint, with more robust failure recovery.
- Evaluate the existing adapters against the full protocol, so every stage—even disappointing ones—appears on the scoreboard.
- Longer term: continued pre-training on Welsh on the cluster. The corpus—roughly 850 million to one billion unique tokens of modern Welsh, before including the National Library's newspapers—has already been collected and tokenised.
This incomplete work is published to record the method, problems and lessons, not just positive results. Fine-tuning a 120-billion-parameter model for cynghanedd is difficult, and describing that difficulty can help further research.