zxell.ai Roadmap (clean copy v2)

Date: 2026-08-12 (revised: 2026-09-05 = section 10, 2026-10-04 = section 11) Original: the roadmap planning document (2026-08-10) This document is a clean copy of the original, updated with the decisions made afterwards (the review01–08 exchanges) and the values measured in phase 0.

1. Project goals and success criteria

1.1 Goals (three layers)

1. Model layer: build a language model from scratch, trained on a news corpus (English, German, French, Japanese; about 8 million articles)(no pretrained models. The model is implemented in-house; using a framework such as PyTorch is allowed — decided).
2. Distribution layer: spread the work across several low-spec clients so that training works without a high-end GPU.
3. Knowledge layer: experience and measure "how intelligent did it become", and obtain quantitative knowledge about the relation: data volume × compute × distribution scheme → capability.

The deliverable is not only "the model itself" but also "quantitative knowledge of how far one can get on a low budget in a distributed setting". The comparison target (a single-machine baseline) and the measurement methods are built into the design from the start.

1.2 Success criteria

2. Decisions (decision log)

The policies decided in review01–08. All later design assumes these.

3. Quantitative assumptions (updated with phase-0 measurements)

3.1 Measured corpus values (inventory of 2026-08-11. Details: the phase-0 data inventory report)

• 7,986,288 articles in total / about 7.57 million effective after link deduplication
• 26.2G characters of body text (≈ 24.8G characters after deduplication)
• Estimated ≈ 6.5G (6.5 billion) tokens (≈4 characters/token for the European languages, ≈1 character/token for Japanese)
• Language mix: en 49.9% / de 25.5% / fr 14.0% / ja 10.6% (but by character count ja is 1.5%, because the body averages only 472 characters)
• Missing bodies are under 1% overall, which is good. category is missing for 100% of ja and 57% of en (classification-type tasks would centre on de/fr)
• 11,069 rows with a newline and 301 with a tab in link/guid → normalization is required in preprocessing

The original estimate (6–9B effective tokens) is confirmed by measurement at 6.5B. No change to the working hypothesis is needed.

3.2 Token count → model size

• By the Chinchilla rule (parameters ≈ tokens ÷ 20), the compute-optimal size for 6.5B tokens is ≈300M.
• The final target L: 500M is somewhat larger than compute-optimal, but multiple epochs (2–3 passes) are common with small data and within the acceptable range. The policy (three sizes) is kept:

Multilingual BPE with a 48k–64k vocabulary; the embedding and the LM head share weights.

3.3 Compute re-estimate (assuming GTX 1050 2GB × 2) [revised downward from the original]

A GTX 1050 peaks at ≈1.8 TFLOPS in FP32 and has no Tensor Cores. At 30% MFU that is ≈0.5 effective TFLOPS per machine (about 1/8 of the original's assumption of "a mid-range GPU with 4 effective TFLOPS"). Training FLOPs ≈ 6 × parameters × tokens.

On top of that, the 2GB VRAM constraint:

• S (30M): comfortable even with fp16 + AdamW. The main target for the two machines on hand.
• M (125M): weights + optimizer state nearly use up 2GB. With gradient checkpointing + an 8-bit optimizer (or moving optimizer state to the CPU) + small-batch gradient accumulation, it "fits but is slow".
• L (500M): cannot be trained in 2GB (inference is possible at fp16 ≈1GB).

Conclusion: with the two machines on hand, run S to completion and go as far as demonstrating the distribution scheme. The full comparison experiments on M and the production training of L presuppose more GPU clients (several with 6GB VRAM or more). This further strengthens the conclusion of section 2.3 of the original, "L is unrealistic without distribution", and the machine count is written into the gate condition between phases 3 and 4 (section 6).

3.4 Communication volume

The gradient of a 500M model is ≈1GB in fp16. Synchronous data parallelism does not work over a home connection (as in the original). The policy stays: H local steps → send the difference (pseudo-gradient) at low frequency, with 8-bit quantization + top-k sparsification to compress it to 1/100–1/500, synchronizing once every 30–60 minutes.

4. Architecture

4.1 Overall structure (hub and spoke. Updated to what is actually running)

[Central server]  https://api.zxell.ai
  (omitted: how the public route is set up)
  ├─ Coordinator API (FastAPI): registration approval / task leasing / receiving results
  ├─ Parameter server: weight_versions (with checksums) + outer optimization (to be implemented)
  ├─ Storage (shards, weights, artifact Δ)
  ├─ PostgreSQL 16: clients / tasks / results / weight_versions
  └─ Dashboard (to be implemented. Based on /api/status, with an admin key)

[Clients] (for now two Windows machines, GTX 1050 2GB)
  ├─ GPU worker: GET weights → GET shard → H local steps → POST Δ
  └─ CPU worker: preprocess / eval / verify tasks

4.2 Server implementation status (the 8 items of section 3.6 of the original → all implemented and committed)

Leased task hand-out (FOR UPDATE SKIP LOCKED, automatic re-queue on expiry, max_attempts), persistent results (metrics JSON + artifact files), approval-gated client authentication (X-API-Key) + admin API (X-Admin-Key), settings via environment variables (pydantic-settings, required), Pydantic v2, weight versioning and distribution, shard distribution. Task types are preprocess / train / eval / verify. A train result requires base_weight_version (the basis for handling staleness). It has passed a smoke test of 15 items plus 2 missing-auth-header items against a real PostgreSQL.

4.3 Distributed training scheme (keeps the recommendation of section 3.2 of the original)

The DiLoCo style (Local SGD + outer Nesterov momentum) is the main candidate, introduced in stages: an MVP with FedAvg (plain averaging) → outer optimization → Δ compression (8-bit / top-k). How quality changes as the scheme is swapped is itself knowledge this project produces. The non-IID countermeasure (random shard assignment so that every client sees every language) and the staleness countermeasure (attach the base weight version; decay or discard) are also as in the original.

4.4 Client v1 (next to be implemented)

• Target environment: native Windows (Python + PyTorch CUDA). The GTX 1050 (Pascal, sm_61) is supported by current PyTorch CUDA builds.
• Loop: register → poll while waiting for approval → fetch a task → process → submit the result. Retries and a heartbeat within the lease period are implemented (omitted: a note about the network line).
• At first registration the client self-reports a simple benchmark (GPU/CPU, VRAM, line) and the task type and H are adjusted.
• Result verification (outlier detection on the Δ norm → spot-check double computation → trust score) is implemented in phase 3.

5. Data pipeline (reflecting what phase 0 found)

1. Normalization: remove newlines and tabs from link/guid (11 thousand measured), normalize the character encoding, check for HTML residue.
2. Deduplication: link as the primary key (5.3% duplicates), guid as auxiliary. Near-duplicate removal with MinHash/SimHash, as a countermeasure against duplicated news-agency copy, will be decided after measuring its effect in phase 1.
3. Concatenation for Japanese: for ja only, concatenate title + description + content_text (decision 11).
4. Tokenization → sharding: multilingual BPE (48k–64k) → fixed-length packing (1024/2048) → shards of tens to hundreds of MB + checksum + metadata (language mix, token count). Location: the server's storage.
5. Split: take time into account (put the final few months into test so that "summarize a new article" is evaluated naturally).
6. Distribute the preprocessing itself as preprocess tasks, as the first real operational test of the distributed infrastructure.

6. Phase plan (position when written: phase 0 complete. See section 11 for the latest)

Phase 0: quantitative understanding and legal groundwork — ✅ done (2026-08-11)

• Data inventory complete (phase0_inventory.md). The legal policy is settled as closed operation (decision 1).
• Server rework and publication (https://api.zxell.ai) (omitted: operational details) completed (ahead of schedule).

Phase 1: single-machine baseline (3–6 weeks) ← next

• Tokenizer training (multilingual BPE 48k–64k, including verification of compression efficiency for Japanese)
• Preprocessing pipeline v1 (section 5) → shard generation
• Train the S model (30M) on a single GPU (GTX 1050) and record the baseline loss curve and generation quality
• Evaluation harness (perplexity, ROUGE, QA skeleton)
• Decision point: S generates reasonable text and evaluation runs automatically
• * Note on the execution environment: the baseline runs on a machine planned as a client (Windows/GTX 1050). The server machine has no GPU, so the division of roles — training on the client side, aggregation on the server side — is needed from the start

Phase 2: distributed infrastructure MVP (4–8 weeks)

• Client v1 (4.4) / operations dashboard v1 (with an admin key)
• Succeed in distributed training of the S model with FedAvg on two machines
• Decision point: training continues with one machine down, and reaches a loss equivalent to single-machine training

Phase 3: low-communication scheme and comparison experiments (4–8 weeks)

• DiLoCo style (outer Nesterov) + Δ compression (8-bit / top-k), verification mechanisms (outliers, recomputation, trust score)
• Compare schemes on the M model → interim report
• Decision point: quality degradation within 10% at 1/100 the communication, and a realistic prospect of securing several GPU clients with 6GB+ VRAM (an additional gate from the re-estimate in 3.3)

Phase 4: production training and the experience side (6–12 weeks)

• Distributed training of the L model (500M) on all data (checkpoints saved regularly)
• Summarization SFT (article + description pairs), inference API and demo UI, QA evaluation, checkpoint archaeology, visualization of model internals (an experience UI in the lineage of "Spreadsheets are all you need")
• Decision point: measurement of the success criteria (1.2) is complete

Phase 5: publishing what was learned and deciding on extensions

• Technical report (comparison of distribution schemes, scaling curves, lessons about multilingual training)
• Decisions on extensions: an open call for participants (if the legal questions are settled), image support (a main image actually exists for 90% of the articles — measured in phase 0), browser (WebGPU) clients, and so on

7. Risks (the main ones. Updated from section 5 of the original to match the current state)

8. Publication policy (decision 6 in concrete terms)

• Make GitHub zxell-ai/zxell public. Only the code is published: zxell_server/, zxell_client/ (to come), README (English), CLAUDE.md, LICENSE (GPL-3.0).
• The private folder (plans, reviews, reports, raw data statistics) stays outside Git (stated in .gitignore). Weights, shards, and the DB are not published.
• It has been confirmed that the commit history contains no real credentials (the DB URL in the initial commit is a placeholder only).

9. References

DiLoCo / Local SGD / FedAvg / Branch-Train-Merge / Hivemind / Petals / BOINC / Chinchilla / nanoGPT / Spreadsheets are all you need (same as section 8 of the original).

10. Revision of 2026-09-05: move to a new server and a change in the client line-up

10-1. What happened (review16–18 and after)

• A physical RAM fault on the old server (confirmed in review17) → the move to a new server, AMD Ryzen 3 4300U / 16GB, is complete. It passed 4 passes of memtest. (omitted: the list of restored services)
• (omitted: details of the old server's disk failure) The live MySQL was lost. However, for the original corpus (the backup of 2025-03-30), the backup copy survived with a fully matching sha256. From here on this file is the source of truth (plus a second copy).
• Decision 8 ("read MySQL directly") has lapsed. The data source is now solely the snapshot (verified JSONL) built by parse_dump.py. extract_sample.py has also been changed to read the snapshot.

10-2. Client line-up (update to decision 2)

Two machines: RTX 3060 (12GB, Ubuntu Server) + GTX 1050 (2GB, Windows). (One of the 1050s was retired and a 3060 obtained in its place. The 3060 has also passed memtest.)

• The phase-3 gate condition "secure clients with 6GB+ VRAM" is already met. Production training of L (500M) is now on a realistic schedule.
• The main arena for distributed experiments is the S scale (because of the 1050's VRAM). 3060:1050 is an extremely heterogeneous pair — a speed ratio of ~15:1 and a VRAM ratio of 6:1 — which makes "DiLoCo with a different number of local steps H per client" a research topic close to real distributed environments (BOINC-like). For M/L the 3060 does the training, and the 1050 handles the S distributed experiments + eval/preprocess.

10-3. Rebuilding phase 1 (as of 2026-09-05)

Everything the faulty-RAM machine produced under work/ was discarded and rebuilt on the new server:

• Snapshot regenerated: 7,969,236 rows, passed post-write verification (statistics match the 8/13 run exactly)
• Sample extraction, training and comparison of the 48k/64k tokenizers: done. 48k adopted (estimated 6.14B tokens). chars/token matches the 8/13 results to the third decimal place (reproducibility of the numbers confirmed)
• Measured split boundaries: the corpus's pub_date ends at 2024-02-12 (tapering to about 2,700 articles/month at the end). test = from 2023-11-13 / val = from 2023-10-14. test ≈ 7 thousand articles, val ≈ 2.7 thousand articles
• Full tokenization + sharding: running

Next: implement the S (30M) model → single-GPU baseline on the 3060 (the completion check for phase 1).

11. Revision of 2026-10-04: phase-1 preprocessing finalized and the S model implemented

11-1. Shards v2 — a new split method and deduplication for Japanese (2026-09-27, review19–21)

• In v1, built on 9/5 (125 shards, 6.152B tokens; one chronological boundary for all languages), val/test were 100% Japanese. The cause was a difference in when collection ended: en/de/fr stopped on the same day, 2023-05-28, while ja alone continued until 2024-02-12. The boundaries in 10-3 (test = from 2023-11-13 / val = from 2023-10-14) have lapsed
• The split was changed to per language, by article count (decision 16, compute_boundaries.py). val/test are now the same size for all four languages, and train loses only 0.3–1.3% per language

• Deduplication in the ja concatenation (textprep.compose_ja): the title or description is not concatenated when it duplicates the start of the body. The title was duplicated in 87.5% of ja articles. ja characters −12%; train tokens 0.23B → 0.197B (about −14%)
• HTML tag removal and entity decoding were added to clean() (bodies containing tags are a small 0.005%)
• Result of v2 (verified. Spot-checked sha256 values match, zero anomalies):

• Caveat: the val/test period for en/de/fr is narrow (the en test set is the last 4 days). If the S model's evaluation looks off, switch to widening the period and sampling up to an article cap

11-2. Implementation of the model (2026-09-29, review22–23)

The model and its training, evaluation, and generation code were implemented in zxell_model/ (RMSNorm, RoPE, causal self-attention, SwiGLU, weight sharing between the embedding and the LM head. AdamW, warmup + cosine, bf16, checkpoint/resume, per-language perplexity evaluation).

• Passed a smoke test of a tiny configuration on the server machine (CPU): train → resume → per-language PPL evaluation → generation
• Even in phase 1, shards are fetched through the API (GET /api/shards/{name}) rather than copied locally, and their sha256 is verified (fetch_shards.py), so that the distribution route is exercised from the start. Demonstrated
• The phase-1 baseline is changed to run on the RTX 3060, from what section 6 says (GTX 1050)

11-3. Plan for the S baseline and the path to phase 2 (decision 18)

In phase 2 the same S is trained with FedAvg on two machines (3060 + 1050), distributed through the API. Processing one shard (50 million tokens) takes about 21 minutes on the 3060 and about 5 hours on the 1050, which exceeds the lease (1 hour), so tasks for the 1050 are cut finely by token range within a shard.

11-4. Publication (carrying out decision 14)

• 2026-10-02: the development log (https://zxell.ai/devlog/) was published
• 2026-10-04: the per-phase reports (https://zxell.ai/reports/, English and Japanese) were prepared. Each component's README now comes in English and Japanese

11-5. Current position (2026-10-04): phase 1 in progress

Remaining: set up the 3060 machine → S baseline training (about 1 day) → check per-language PPL on test and generated samples → the phase-1 completion check ("S generates reasonable text and evaluation runs automatically") → phase 2 (replace the client's train task with real local training and run FedAvg on two machines).