Sebastian Raschka engagement report

Best tweets

• Jun 30, 20265.0x their median

  After 18 months of writing, coding, and experimenting, Build a Reasoning Model (From Scratch) is finally out! My first copies just arrived! 📚 440 full-color pages. Inference scaling, reinforcement learning, and distillation from scratch. https://t.co/647ksI7sLc

  6.0K55928153533K viewsView on X
• Jul 10, 20264.1x their median

  For agentic coding, one can say: - Unless you need Terra Ultra perf, it's always better to use a Luna model with higher effort setting (same or better performance but cheaper). - Forget everything below Sol High, use Luna with higher effort settings here - Forget Sol Extra High, use Terra Ultra here - The extra cost of Sol Ultra is probably not worth it over Max

  4.7K458329193973K viewsView on X
• Jul 18, 20263.9x their median

  How can an LLM switch between low-, medium-, and high-effort reasoning? And how does an LLM learn to reason more or less? I put together a “little” article explaining how these effort levels are implemented at inference time and during training. https://t.co/mc4qiCnq0C

  4.5K68010847323K viewsView on X
• Jul 28, 20263.5x their median

  The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.

  4.1K62010530237K viewsView on X
• Aug 7, 20263.4x their median

  Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks also to everyone who shared ideas and opened PRs with improvements! Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories). I am also currently working on a larger applied custom “small” LLM project. It has been keeping me super busy this month, but I will share more on that soon. If you are new to it, some of the highlights include 1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.) 2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts). 3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts 4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA So, if you don’t have any weekend plans yet, happy tinkering!

  4.0K5598126170K viewsView on X
• Apr 26, 20262.6x their median

  April was a pretty strong month for LLM releases: - Gemma 4 - GLM-5.1 - Qwen3.6 - Kimi K2.6 - DeepSeek V4 All are now added to the LLM Architecture Gallery. More details once I am fully back in May! https://t.co/HDYbWi2pcc

  3.0K4347625128K viewsView on X
• Jul 28, 20262.5x their median

  Yes, LLM architectures are getting a little more complicated https://t.co/jII112pJbO

  3.2K1559122175K viewsView on X
• Jul 29, 20262.5x their median

  We ran Kimi K3 through 3 agent harnesses (Claude Code, Hermes, Kimi Code) on 28 identical tasks. All 3 harnesses completed the tasks at similar success rates, but the interesting story is token efficiency: the same task cost up to 30x more tokens depending on the harness. 🧵🧵

  2.9K177219103687K viewsView on X
• Jun 16, 20262.4x their median

  WHAT THE HELL is happening in AI? A 3B parameter model just put up coding benchmark scores in the same league as Claude Opus 4.5. 3 BILLION. The weights are on Hugging Face, anyone can test it. I genuinely don't know if this is a breakthrough or if the benchmarks are broken. https://t.co/8nVIbwjLUQ

  2.8K18220884680K viewsView on X
• May 20, 20262.3x their median

  Introducing: Cohere Command A+ We’ve created our most powerful LLM yet, optimized it to run on as little hardware as possible, and released it open-source for all. https://t.co/C1KYnvA8JB

  2.6K367103113743K viewsView on X

Ranked by total interactions across everything we have tracked for this account, which is a longer history than the 30-day window the rates above use. The multiple compares each post to this account's own median.

Keep going