Phase 1: File layout and data formats on the client (the RTX 3060 machine)

Date: 2026-10-11 Scope: the files created or placed on the client machine in the course of the production S baseline run (started 2026-10-10, completed all 90,000 steps). The numbers and structures were taken from the actual files as copied to the server. This is also the basis of the input/output specification for phase 2, when client v1 is replaced with real local training.

1. Directory layout (the 3060 machine, under the user's home)

~/
├── zxell/                              git clone (the public repository zxell-ai/zxell, main)
│   └── zxell_model/
│       ├── .venv/                      Python venv (torch cu128, numpy, sentencepiece)
│       ├── config.py, model.py, data.py, train.py, eval.py, sample.py, fetch_shards.py
│       └── runs/                       training output (gitignored)
│           ├── S_check/                the 30-step check run (disposable: config.json / log.jsonl / ckpt_last.pt / ckpt_final.pt)
│           └── S_baseline/             ★ the production run
│               ├── config.json         model configuration + training arguments + the list of shards used
│               ├── log.jsonl           training log (9,181 lines)
│               ├── stdout.log          tee of standard output (same content as log.jsonl + the preset line at the top + the done line)
│               ├── ckpt_last.pt        resume checkpoint, overwritten every 1,000 steps (finally at step 90000)
│               └── ckpt_final.pt       checkpoint at the end (349MB) ★ the deliverable
├── zxell-shards/                       ★ training data (fetched through the API, 231 files, about 11GB)
│   ├── shards_index.json               list, boundaries, metadata
│   ├── train_000.bin … train_111.bin   training shards (112 of them, each ≤ 50 million tokens ≈ 100MB)
│   ├── train_NNN.bin.json              metadata of each shard (112 files)
│   ├── val_000.bin / val_000.bin.json / val_000.langs.u8      validation (11,129 articles)
│   └── test_000.bin / test_000.bin.json / test_000.langs.u8   test (30,171 articles)
└── sp_bpe_48k.model                    tokenizer (1,035KB. For decoding in sample.py. Copied from the server with scp)

2. Training data (~/zxell-shards/)

2-1. shards_index.json — list and metadata

Written by the shard step of tokenize_full.py. fetch_shards.py fetches this first and verifies each shard against the sha256 listed in it.

{
  "tokenizer": "sp_bpe_48k.model",
  "dtype": "uint16",
  "shard_tokens": 50000000,
  "seed": 20260813,
  "created": "2026-10-10T01:05:23",
  "boundaries": { "mode": "per_lang_count", "test_docs": 8000, "val_docs": 3000,
                  "computed_at": "...", "snapshot": "...",
                  "langs": { "en": { "docs": 3978459, "max_pub_date": "2023-05-28T02:30:00",
                                     "test_start": "2023-05-24T11:32:11", "val_start": "2023-05-23T08:00:00",
                                     "test_docs_actual": 8000, "val_docs_actual": 3007 }, "de": {...}, "fr": {...}, "ja": {...} },
                  "_meta": {...} },
  "shards": [ { "name": "train_000.bin", "split": "train", "tokens": 49999705, "docs": 66538,
                "sha256": "caa2269f…", "langs": { "en": 30775576, "de": 12011218, "fr": 5450408, "ja": 1762503 } },
              ... 114 entries ... ]
}

2-2. <split>_NNN.bin — the token sequence itself

• Format: a raw array with no header. uint16 little-endian, 1 element = 1 token ID (0–47,999). The vocabulary of 48,000 is below 65,536, so 2 bytes are enough
• Article boundaries: one EOS (ID 2) is placed at the end of each article. The i-th article ends at the i-th EOS (no BOS is inserted. The special tokens are <unk>=0, <s>=1, </s>=2)
• Order: train is shuffled as a whole at the article level and then cut every 50 million tokens at article boundaries (no article spans two shards). Every shard has the four languages mixed in nearly the same ratio as the whole corpus (for example, train_000 is en 61.6% / de 24.0% / fr 10.9% / ja 3.5%). val / test are not shuffled and stay in scan order (the snapshot's id order. Articles from the same channel cluster together)
• Size: train_000.bin = 99,999,410 bytes (49,999,705 tokens). val 11.4MB, test 31.6MB
• Example of the first 8 tokens (train_000): 27104 1625 38641 576 263 10258 307 37908

How to read it (Python): np.memmap(path, dtype=np.uint16, mode="r") (load_shard in data.py). The training sampler "picks a shard in proportion to its token count and cuts ctx+1 tokens from a random position" (crossing article boundaries is allowed).

2-3. <split>_NNN.bin.json — shard metadata

{ "name": "train_000.bin", "split": "train", "tokens": 49999705, "docs": 66538,
  "sha256": "caa2269fbc07182deffdb94f05d39ccd7771e93cd07de971aa3dd50e6548b7d2",
  "langs": { "en": 30775576, "de": 12011218, "fr": 5450408, "ja": 1762503 } }

sha256 is the hash of the whole .bin (used both for the read-back verification right after writing and for the check after fetching). langs is the token count per language (including EOS).

2-4. val_000.langs.u8 / test_000.langs.u8 — language of each article

• Format: 1 byte per article, in scan order. Values are 0=en, 1=de, 2=fr, 3=ja (the order of config.LANGS)
• Size = number of articles: val 11,129 bytes, test 30,171 bytes (checked against docs in the .bin.json when the files were placed)
• Purpose: per-language perplexity. During evaluation the EOS tokens in the token sequence are accumulated to find "which article this token belongs to", and this array gives the language (data.token_langs). train has no such file (no per-language evaluation is done on train)
• Example of the first 16 bytes (val): 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 (val begins with a run of French articles)

3. Training output (runs/S_baseline/)

3-1. config.json — the full set of settings needed for reproduction

Written by train.py at start-up.

{
  "model": { "n_layer": 6, "d_model": 384, "n_head": 6, "ctx": 1024, "vocab_size": 48000,
             "rope_theta": 10000.0, "tie_embeddings": true },
  "args":  { "preset": "S", "shards": "<home>/zxell-shards", "out": "runs/S_baseline",
             "steps": 90000, "batch": 32, "grad_accum": 1, "ctx": 0,
             "lr": 0.0006, "min_lr_ratio": 0.1, "warmup": 500, "wd": 0.1, "clip": 1.0,
             "eval_every": 500, "eval_tokens": 1000000, "final_eval_tokens": 0,
             "ckpt_every": 1000, "log_every": 10, "device": "auto", "seed": 20260929,
             "resume": "", "max_train_shards": 0 },
  "train_shards": [ "train_000.bin", ... , "train_111.bin" ]
}

3-2. log.jsonl — training log (one JSON per line, two kinds of record)

Training records (every 10 steps + step 1. 9,001 lines):

{"step": 1,     "loss": 10.864,  "lr": 1.2e-06, "grad_norm": 3.206, "tok_per_s": 33597, "elapsed_s": 1}
{"step": 90000, "loss": 3.1602,  "lr": 6.000000016633646e-05, "grad_norm": 1.516, "tok_per_s": 38298, "elapsed_s": 77004}

Evaluation records (every 500 steps + at the end. 180 lines):

{"step": 500,   "eval": "val",      "all": 384.59, "en": 186.68, "de": 367.7,  "fr": 221.68, "ja": 2635.84}
{"step": 90000, "eval": "val_full", "all": 32.33,  "en": 20.18,  "de": 28.88,  "fr": 21.8,   "ja": 132.2}

On a resume (--resume), records are appended to the same file (no resume this time. The two start-ups on 10/9 were removed with rm -rf before starting).

3-3. ckpt_last.pt / ckpt_final.pt — checkpoints (349MB)

A dict saved with torch.save. Read by eval.load_model() and train.py --resume.

Tensor names in model (one block as an example):

tok_emb.weight                 (48000, 384)
blocks.0.norm1.weight          (384,)
blocks.0.attn.qkv.weight       (1152, 384)     ← Q/K/V in a single matrix
blocks.0.attn.proj.weight      (384, 384)
blocks.0.norm2.weight          (384,)
blocks.0.ffn.w_gate.weight     (1024, 384)     ← SwiGLU (hidden 1024)
blocks.0.ffn.w_up.weight       (1024, 384)
blocks.0.ffn.w_down.weight     (384, 1024)
... blocks.1–5 likewise ...
norm_f.weight                  (384,)
lm_head.weight                 (48000, 384)    ← shared with tok_emb.weight

• ckpt_last.pt is overwritten every 1,000 steps (a safeguard against interruption). ckpt_final.pt is saved under a separate name at the end
• What phase 2 handles in weight distribution and Δ exchange is only the model state_dict (optimizer state stays local to each client). 116MB in fp32, 58MB if reduced to bf16. Because of weight sharing, lm_head.weight need not be sent (the receiver ties it back to tok_emb.weight)

3-4. stdout.log

Standard output captured with tee. Its content is the same lines as log.jsonl, plus, at the top, preset S: 29.1M params (10.6M non-embedding), ctx 1024, device cuda and, at the end, done: 90000 steps, 2949.12M tokens, 77050s. Tracebacks from failures also remain here (as with the OOM — review24).

4. Sizes

5. What can be reproduced from what

6. What is kept where

• The 3060 machine: everything in runs/S_baseline/ + ~/zxell-shards/ (left in place, as agreed in question 4 of review26)
• The server: a copy of the deliverables (config.json / log.jsonl / ckpt_final.pt) and the source the shards are distributed from
• Checkpoints are not published (decision 6). .gitignore already keeps runs/ and *.pt out of GitHub