ResearchNovember 17, 2026
13
🌱

Tree-grams Challenge Transformer Dominance

Tree-grams: hierarchical n-gram models rival transformers on language modeling.

#n-gram#language modeling#interpretability#muP
N-Gramme schlagen Transformer mit Baumtrick
Share Article
šŸ”„ What happened Researchers on arXiv introduced "tree-grams": autoregressive n-gram models that build conditional probabilities through a hierarchical composition of order-2 and order-3 tensors. Two hyperparameters — context window n and rank d — control model capacity. The team shows tree-grams beat classic n-gram baselines while staying competitive with deep transformer language models. šŸ’” Why it matters Explicit control over horizon and capacity lets you isolate effects that blur inside transformer black boxes. The kicker: when allowed to route across candidate topologies, tree-grams develop emergent sparsity — up to 95% of topologies can be pruned without meaningful performance loss. That's a rare case of interpretable scaling. ⚔ Our take Maybe the transformer was never the endgame — just a detour. If you actually want to understand language models, stop building bigger black boxes and start taking controlled approximators like this seriously.
The title, summary and analysis of this item were produced automatically by an AI system and have not been editorially reviewed. They may contain errors, bias or omissions — when in doubt, read the linked original source.