What it does
BYTEFRAME removes the tokenizer entirely. Every byte value enters the model through a fixed 256-by-d_model matrix whose rows are spectral signatures, a pure function of the configuration, carrying no learned parameters, registered as a non-trainable buffer.
Three projection geometries span the ablation: a discrete Fourier basis, sequency-ordered Walsh-Hadamard functions, and band-limited frames from a checksummed wavetable. Each one is injective, and the repository states exactly which component guarantees that.
In the code
"""Deterministic spectral projection of byte identity.
BYTEFRAME hard constraint: no tokenizer, no vocabulary table, no learned
embedding lookup. Byte identity enters the model through this module: a
fixed (256, d_model) matrix whose row b is the spectral signature of byte
value b. It carries no learned parameters; the model registers it as a
non-trainable buffer, and indexing it is mathematically identical to
applying the projection to a one-hot byte.
"""
SPECTRAL_KINDS = ("dft", "walsh", "wavetable")
# All kinds computed in float64, row L2-normalized, cast at the end,
# and use no RNG anywhere.No RNG anywhere in the projection. Two runs on two machines produce bit-identical embedding matrices, which is what makes the ablation between the three geometries meaningful rather than noise.
How this differs from the ordinary version
No vocabulary means no vocabulary problems
Tokenizers create out-of-vocabulary behavior, language bias toward whatever the training corpus over-represented, and a fixed unit of meaning chosen before the model is trained. Operating on raw bytes removes all three, which matters directly for bilingual and mixed-script text.
The embedding is derived, not learned
Because the projection is a pure function, it costs no parameters, cannot overfit, and is identical across runs. The ablation asks which geometry is best, a question you can only ask when none of them are being trained.
In the field
Why bytes are interesting on this border
Text here is Spanish and English in the same sentence, with part numbers, units and OCR noise mixed in. Tokenizers handle that badly and their failures are invisible. A byte-level model has no vocabulary to fall out of.
Questions
- Is this competitive with a production model?
- No, and it does not claim to be. It is an architecture experiment with a documented journal of what each ablation showed.