TinyStories Data
The token pipeline — TinyStoriesV2 validation text, BPE-encoded llama2.c-style, byte-reproducible from the public dataset.
TinyStories Data
All fine-tuning runs draw from one flat token
array, content/data/tokens.npy: the public
TinyStoriesV2
GPT-4 validation split, BPE-encoded with the llama2.c tokenizer
(prepare.py), each story prefixed with BOS, stored as uint16. Training
batches are random T=512 windows; validation is the last 5% of tokens.
The committed token array is reproducible from the public dataset: the
build re-downloads the source text, re-tokenizes a
2000-story prefix with the
committed pipeline, and confirms it matches
tokens.npy byte-for-byte
(prefix match = yes,
total = 5885589 tokens).
A full re-tokenization (27,630 stories, 5,885,589 tokens) was reproduced
byte-identically in the authoring environment (matching sha256), but exceeds
the build's per-check time budget, so only the prefix is re-verified
continuously. verifiedHow we know this
| stories=2000 | prefix_tokens=424918 | prefix_match=yes | total_tokens=5885589 |
|---|
The tokenizer port itself is exercised transitively everywhere: the Java and PyTorch implementations share tokenizer files and were verified for encode / greedy-generation parity, and the round-trip check would surface any drift as incoherent generation or CE mismatch.