Reddit r/LocalLLaMA

my laptop is very old, like 20G of ram 4G Vram 256G NVMe Gen3 From samsung i7 8th gen on these specs i got around 0.67 - 0.77 token per second yes, extremely slow, most of you won't even like it, or use it but free APIs these days are rare, and extremely limited usage, and my country has banned worldwide transactions, so No Claude, No GPT, No Argon my only way is Local LLMs, and here we are i achieved some projects with Qwen, actually, it fixed bugs in my old projects, i used it for modding games (like grey hack, Cell to Singularity, Some of itch io games, using Dnspy and Ghidra) it done pretty well, some tasks required iterations i used LM Studio, and for coding i used Pi agent mostly, and sometimes Opencode i genuinly loved that model, but lately i have been interested more in Ornith 1.5

As of November 2026, what is the best harness for doing long autonomous coding tasks, ones that have full loop of dev/test/update etc cycles, things that might take days+ to complete? So for example, let's say you have 200 pages of some random test game to build, you feed your harness a blender MCP, image gen API, aUnigen/Godot/Unity MCP etc., with the task likely taking 8 to 9 figures of tokens from a local LLM... What would you say is the best at this point? Thanks submitted by /u/StartupTim [link] [comments]

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan. It runs a lot of Qwen models by now: - Qwen3.5 / 3.6: 2B up to 35B-A3B - Qwen3: 4B dense, 30B-A3B - Qwen3.8-27B dense - Qwen3-Next-80B-A3B - Qwen3.8-Flash-Next Quant support goes from IQ1_S to Q8_0, plus BF16 and F16. The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs. My setup is a 3060 12 GB, a 7600X and 32 GB RAM. Qwen3.8-Flash-Next This was the most interesting one. Flash-Next is a strange model: 4 hyper-connec

Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right? What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently. Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and i

Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to contin

Sipeed has recently shared additional details about the MaixStation, a Linux AI workstation powered by Axera’s AX8860P processor, which combines a 16-core Arm CPU with an NPU rated at 256 TOPS FP4. The system supports up to 128GB of LPDDR5X memory, 273GB/s memory bandwidth, 10G SFP+ networking, and PCIe 5.0 expansion for local AI inference and development. https://linuxgizmos.com/pocket-sized-linux-ai-box-runs-on-256-tops-axera-chip/ submitted by /u/DeliciousBelt9520 [link] [comments]

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone. It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half). I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no

I finally got around to trying it. This tiny model is surprisingly capable it seems. Also tried the 2b VL - also very good but it has half the context of 1.7b. It's also quite fast: $ llama-bench -m LFM3-1.7B-Q8_0.gguf -fa 1 ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB): Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes, VRAM: 24112 MiB | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | lfm3 1.7B Q8_0 | 1.71 GiB | 1.73 B | CUDA | -1 | 1 | pp512 | 20648.61 ± 1786.65 | | lfm3 1.7B Q8_0 | 1.71 GiB | 1.73 B | CUDA | -1 | 1 | tg128 | 329.03 ± 2.05 | build: 28b812d6c (12087) This makes me want to try their omni model for wat

From the article : On computer use, Tencent’s table puts Hy4 close to Claude Opus 5.5 and Kimi K3.1 Pro. submitted by /u/mawcopolow [link] [comments]

Hardware : 4× NVIDIA RTX 3090 , all pcie x16 256GB DDR4 Engine : Maya https://github.com/mw00/project-maya model : orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF ( mix-256G) at 524k context Prefill : 400 – 600 tok/s Decode : 30 – 40 tok/s (peaking up to 60 tok/s) ( with dspark ) Hi, everyone i'm still playing around with the model comparing it mostly to qwen 3.8 next flash (Q8 ) , and trying to keep myself busy until we get qwen 4 .So far i find the model doesn't loop as much as the previous version (0731), i also tried to run it at Q8 but it's really slow ( 6-8tok/sec decode , 65tok/s prefill ) .I still prefers to use qwen just for the speed i get with agentic task . Curious to hear what kind of speeds others are getting with similar multi-GPU rigs or different engine ( didn't set u

Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations f

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-worl

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B. These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models. blog： https://www.camthink.ai/blog/small-language-model-benchmark/ project： https://github.com/camthink-ai/NeoMind submitted by /u/KienShen [link] [comments]

Hey everyone, Not a native speaker, happy to be corrected! Can only learn from it. Back when I started out in march 2025, you had: llama.cpp ollama (repackaged llama.cpp for ease of use) koboldcpp (repackaged llama.cpp for roleplay) vllm (for professional deployments) exl3 + tabbyapi (for bleeding edge) litert (for mobile devices) Nowadays everyone seems to be making their own; Strata, Basalt, Maya, Colibri, Ninfer, Infernix... too many to list and keep up with, and genuinely hard for me to keep up with. Then there is the speed of it all. llama.cpp has incredible improvements each week, somehow we can make Qwen3.8-Flash-Next and DeepSeek-v4.1-Flash-1017 run on 64GB RAM + 32GB VRAM, and the likes on those new inference engines (with a catch, usually bad quality). Personally I'm stuck with l

Basalt 0.2.0: Quality & Speed Hey all! I wanted to share a quick update on Basalt , a Blackwell only inference engine for Qwen 3.8 Flash Next. As usual, speed is the headline but that's not what I want to open this post with, I've been reading lot of discussions talking about quality recently, and the main takeaway from most people (which I agree with) is "speed means nothing if your model isn't smart enough", so this time I want to lead with quality, and all of the new measurements which are run against the full precision model as reference (BF16 weights served using llama.cpp). Every number below uses Unsloth's BF16 GGUF, llama.cpp (reference PPL 3.2749), scored using basalt-bench --quality . The old base was llama.cpp Q8, which funnily enough made things look worse because of its own er

Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all. I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like He had been hiding in the attic for months when the soldiers found him. it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over ( he had lived in Paris for ten years, but by 1990 he had moved back ), the original tense has to stay. So it's not "a

A open source harness that was used with Qwen3.8 FN that QwenDev did a shout out to. Available now. From QwenDev via x https://x.com/QwenDevs/status/2117644199860601297 “Well, turns out Qwen3.8-Flash-Next can do more than work with software tools, it can operate real robots too. A group of Qwen developers from Tsinghua and UC Berkeley recently built uni-robot-agent, an open-source harness that connects general-purpose LLMs to robotic systems. It gives Qwen a way to interact with the physical world through tool calls without changing the model weights. With this harness, Qwen3.8-Flash-Next reached the performance level of some previously reported closed-model robotic agent systems. The average API cost was $1.09 per task, compared with $21.29 for GPT-6 Astra. On 19 real-world robot manipula

I've been working on Turkish ASR for a while and put out v2 recently. Two models, both CTC + KenLM, both meant for CPU (both trained for under $250): - balbal-v2-manas: 313M, Turkish + English, ~123x real time on 8 threads - balbal-v2: 45M, Turkish only, ~236x real time. It even runs faster than real time on one core of an old Celeron POS box running Windows 7. Mu-Bench is a Turkish benchmark of 846 real customer service calls. Numbers from the official harness: Model WER sigWER Google Chirp 3 8.03 14.30 balbal-v2-manas 8.20 14.89 Gemini 3.5 Transcribe 8.68 18.20 ElevenLabs Scribe v2 9.43 19.15 balbal-v2 9.46 18.44 Deepgram Nova-3 13.37 24.35 Azure Speech 14.02 30.73 Qwen3-ASR 1.7B 14.76 30.61 whisper-large-v3-turbo 24.65 47.04 On FLEURS Turkish manas is basically tied with whisper-large-v

MIT CSAIL had GPT-6 Astra fix Qwen3.5-0.8B's reasoning loops by editing 0.016% of its weights (MATH-500 went from 7.9% to 48.1%). They released the diff, so the edit can be rebuilt exactly and run locally. I took it a step further and had Claude look at Qwen with Anthropic's Jacobian lens. At the token where the model decides to write "Wait" again, the lens reads "Wrong", "Nope" and "Hmm" from about 60% depth on, even though the answer is already written out. At real stopping points it reads "</think>", "Answer" and "Output". With the edit applied, the lens at the same token shows "Verification", "Completion" and "Result" in place of the doubt. The edit converts to GGUF cleanly (the base and edited bf16 files differ only in the two edited tensors) and still holds at Q4_K_M, with 1 loop in

Hey folks, About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work. We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!) . The problems we're trying to solve haven't really changed and now some of our customers use it: One inference endpoint dies and your workflows grind to a halt. You've got models spread across different machines, GPUs and runtimes, but no easy way to manage them as one fleet. Switching between Ollama, vLLM, SGLang, llama.cpp or LM Studio means reconfiguring applications. Multiple machines serving the same model, but no unified load balancing o