Phase 0: Data Inventory Report

Date: 2026-08-11 (everything was done read-only, with SELECT statements only)

omitted: where the raw data is stored

1. Overview

Item	Value
Total articles	7,986,288 (no change during the inventory = confirmed that collection has stopped)
Channels	803 defined / 794 with articles
Languages	en / de / fr / ja (decided from the first 2 characters of feed_channels.language)
Article period (pub_date)	1990-05-20 to 2024-04-25 (1990 and the like appear to be anomalies on the feed side. The bulk is 2020–2024)
Body text (content_text) total	26.2G characters
Missing bodies	Under 1% overall (very good)
Channel concentration	The top 31 channels account for 50% of the total (largest: a German-language channel with 362,048 articles; median 1,545)

2. Distribution and quality by language

Language	Channels	Articles	Share	Body characters	Mean body length	Body missing	Summary missing	Category missing	Has img_url
en	408	3,984,113	49.9%	17.08G	4,311 chars	0.6%	5.0%	57.1%	12.1%
de	162	2,036,736	25.5%	6.02G	2,963 chars	0.2%	4.7%	33.7%	51.9%
fr	176	1,116,522	14.0%	2.70G	2,440 chars	0.8%	18.8%	34.5%	26.9%
ja	48	848,917	10.6%	0.40G	472 chars	1.2%	63.2%	100%	0.1%

The most important finding: Japanese bodies are extremely short. A mean of 472 characters is about "headline + lead", and shows that many Japanese feeds do not deliver the full text. As a result Japanese is 10.6% by article count but only 1.5% of the total by character count (395 million characters). This bears directly on how training is apportioned and how evaluation is designed in the first period (→ question 3).

Other notes on quality:

title is varchar(100), so long titles are truncated (known; used as is in the first period)
category is 100% missing for Japanese and 57% for English → training that uses categories (classification tasks and so on) must centre on de/fr, or be designed to tolerate missing values
11,069 rows contain a newline in link/guid, and 301 contain a tab → normalization is required in preprocessing (it also caused misaligned dump rows in this inventory)

3. Duplicates (link / guid)

Measure	Value
Unique links	7,565,028 (94.7% of the total) → 5.3% duplicates by link (421,260 rows)
Empty guid	1,725,457 rows (21.6%)
Unique guids (excluding empty)	5,787,094 / 6,260,831 (7.6% of the rows that have a guid are duplicates)

Preprocessing policy (making the review03 agreement concrete): deduplicate with link as the primary key (→ about 7.57 million effective unique articles). guid is empty in 20% of rows, so it is used as an auxiliary key.

4. Image files

Not used for training in the first period; inventory only (as agreed in review03).

Item	Value
Total files	16,495,782
Main (`{id}_1.*`)	7,470,059
Thumbnail (`{id}_0.*`)	7,198,315
Other naming	1,827,408
By language	en 7.67M / de 4.79M / fr 2.39M / ja 1.63M

7.47 million main images against 7.98 million articles → an image actually exists for over 90% of the articles. This is far higher than the share of rows with feed_items.img_url (12–52%), so when multimodal training is considered in the second period, the right approach is to design around which files actually exist.

5. Estimated corpus size (first period, text only)

Effective articles after deduplication: about 7.57 million
Characters: about 26.2G × 94.7% ≈ 24.8G characters
Rough token count (assuming multilingual BPE: ≈ 4 characters/token for European languages, ≈ 1 character/token for Japanese): European 24.4G characters ÷ 4 ≈ 6.1G tokens + Japanese 0.37G characters ≈ 0.4G tokens → roughly 6.5G (6.5 billion) tokens

This is ample for first-period training of small models (for reference: by the Chinchilla rule, a model of the 300M-parameter class calls for ≈ 6G tokens).

omitted: 6. Infrastructure findings (findings about the server's disk)

7. Questions

omitted: question 1 (a database configuration change) and its reply

Confirming the snapshot policy: measurement has confirmed that collection has stopped, so may I proceed with preprocessing reading feed_items directly, without taking a snapshot such as a mysqldump? (If there is no plan to resume collection, this is the simplest. If there is, I will note max(id) when preprocessing starts and read "id ≦ that value" — either works.)

Architect's replyFor now, no snapshot is fine.

Handling of Japanese (how training is apportioned in the first period): Japanese is only 1.5% of the total by character count, and its bodies are mostly short. My proposal is as follows:
- In the first period, train on all languages mixed as they are (natural ratio. Japanese performance is expected to be limited)
- For Japanese only, concatenate title + description + content_text to gain text per article (other languages use content_text only)
I recommend doing both. Is that acceptable? (If you would like, for example, to prioritize Japanese, upsampling and the like can also be considered.)

Architect's replyOK.

8. Next steps (towards phase 1)

Once the questions are answered: present concrete plans for tokenizer training (multilingual BPE) and shard design
Preprocessing pipeline design: normalization (remove newlines/tabs, check for leftover HTML tags) → link deduplication → language tagging → tokenization → sharding
Client v1 and dashboard v1 (waiting for the replies to the questions in review08)