Introducing Mistral Large 4: Le chonk

❗️Mistral drops a 1T-parameter “Le Chonk” Mistral AI has unveiled Mistral Large 4, internally nicknamed “Le Chonk”, a 1 trillion-parameter natively multimodal model with just 49B parameters active at a time. Mistral says it is currently the best open-weights model from the US or Europe across aggregated benchmarks, while reaching state-of-the-art performance in areas including cybersecurity, manufacturing and finance. It also reportedly surpasses closed frontier models on visual grounding. The model was developed end-to-end in Europe and runs on Mistral’s own European cloud infrastructure. API access is available today, while the open-weights release is planned for the end of October. Mistral is also working privately with cybersecurity partners on the model. Source. @aipost 🏴

UT-based small models, despite being trained from scratch on these tasks without internet-scale pre-training, consistently outperform most standard Transformer-based Large Language models (LLMs) by a significant margin. 👈 The Universal Transformer (UT) extends the standard Transformer by introducing recurrent computation over depth. Instead of stacking L distinct layers, the UT applies a single transition block repeatedly to refine token representations. For an input sequence x with embedding matrix H 0 ∈ R n×d , the UT updates states as H t+1 = LayerNorm(H t + MHA(H t )) followed by a shared position-wise transition function H t+1 ← LayerNorm( H t+1 + Transition(H t+1 )), t = 0, . . . , T − 1 where Transition is either a feed-forward network or separable convolution. To encode both positi