Training a language model from scratch — without the GPU cluster.
An experiment in low-budget, low-communication distributed training: a multilingual news corpus, a from-scratch Transformer, and a swarm of ordinary consumer machines coordinated over HTTPS.
The project
The corpus is a private archive of roughly 8 million news articles (title, summary, category, body) in English, German, French, and Japanese — about 6.5 billion tokens after deduplication. Instead of renting a GPU cluster, a small central server hands out training tasks to low-spec volunteer machines.
The deliverable is not just the model. It is quantitative insight into how far a from-scratch model can get on commodity hardware: perplexity and summarization quality against a single-machine baseline, scaling curves over client count, and the real cost of cutting synchronization traffic by orders of magnitude.
How it works
Hub-and-spoke, in the spirit of volunteer computing (BOINC, Folding@home):
[Central server]
├─ Coordinator API (FastAPI): registration & approval, task leasing, result intake
├─ Parameter store: versioned global weight snapshots (checksummed)
├─ Shard storage: tokenized, packed training shards
└─ Metadata DB (PostgreSQL): clients / tasks / results / weight versions
[Clients] (heterogeneous, low-spec)
├─ GPU workers: GET weights + shard → train H local steps → POST compressed delta
└─ CPU workers: preprocessing / evaluation / verification tasksSynchronous data parallelism is impossible over home connections — a 500M-parameter gradient is ~1 GB per step. Training therefore follows the DiLoCo / Local SGD family: each client trains locally for hundreds of steps and submits only a compressed weight delta; the server aggregates deltas with an outer optimizer and publishes a new weight version. Tasks are leased and automatically re-queued, so clients can join, leave, or fail at any time.
The model
A decoder-only GPT-style Transformer implemented from scratch (PyTorch as the base framework, no pretrained weights): RoPE positions, RMSNorm, SwiGLU FFN, tied embedding / LM head, and a custom multilingual BPE tokenizer (48k–64k vocabulary).
S — ~30M params
6 layers · d=384 · ctx 1024. Pipeline validation and smoke tests; runs on a 2 GB GPU.
M — ~125M params
12 layers · d=768 · ctx 1024. Main testbed for comparing distribution schemes.
L — ~500M params
24 layers · d=1280 · ctx 2048. Final target, trained on the full corpus.
Roadmap
| Phase | Content | Status |
|---|---|---|
| 0 | Data inventory, legal policy, server & public HTTPS endpoint | done |
| 1 | Tokenizer, preprocessing pipeline, single-machine baseline | in progress |
| 2 | Distributed MVP: client v1, FedAvg across machines, dashboard | planned |
| 3 | Low-communication training (DiLoCo-style), verification & trust | planned |
| 4 | Full L-model training, summarization SFT, evaluation & visualization | planned |
| 5 | Public technical report; extension decisions | planned |
Participation & data
The project currently runs closed: clients are machines operated by the project itself, and the news corpus is never redistributed — only tokenized, shuffled, packed shards reach approved clients. The coordination server and client code are open source (GPL-3.0 on GitHub); model weights and data statistics are not published at this stage.