Reddit r/LocalLLaMA

The model is derived from Qwen3.8-27B, a 27B hybrid-attention causal language model (architecture unchanged), but uses ternary weights to shrink model size down to <6GB in size. According to the model card, it's 9x smaller than FP16 while retaining 98.2% of the intelligence. - Collection: https://huggingface.co/collections/prism-ml/bonsai-2 - Demo: https://huggingface.co/spaces/webml-community/ternary-bonsai-2-webgpu-kernels submitted by /u/xenovatech [link] [comments]

IFM released K2-Horizon-7B , diffusion augmented LLM at upto 5200 tps, with claimed lossless speedup. causal LLM architecture and adds a plug-and-play diffusion adapter alongside the autoregressive weights. https://arxiv.org/abs/2609.04010 submitted by /u/Zulfiqaar [link] [comments]

We've been tuning vLLM/SGLang/llama.cpp setups for a long time and got tired of the profiling part: nsys trace, open the GUI, squint, change a flag, repeat. The profilers assume a human is looking at the timeline. These days the thing doing our tuning is usually an agent, and it can't look at a timeline. So: https://github.com/graphsignal/graphsignal It's a sidecar profiler. You wrap whatever you're running: graphsignal-run vllm serve <model> --port 8000 graphsignal-run sglang serve --model-path <model> --port 8000 graphsignal-run ./llama-server -m model.gguf graphsignal-run python whatever.py and it serves everything it measures at http://127.0.0.1:18259/signals as one JSON: time per kernel / per CUDA graph / per memcpy and sync, NVML stuff (util, VRAM, power, clocks, throttling, XID erro

I posted something about a 9b model a few days ago. The real problem was 2 fold. Someone suggested the size was too big to prove it all out. It's a fair argument. I needed the model depth though. The second problem was the "ability" of these models and the fact that labs (with money) produce these models still. No real usefulness in my mind. A 2b model? maybe interesting. Want a chatbot? - this could be it. Further, the Llama licensing at the core of the model was highly problematic and had to be thrown in the trash. It's vastly too restrictive and I couldn't Apache 2.0 anything. So I moved to using the OLMo tokenizer. Except I shrunk the d_model down to 2048 so I could build a tiny 2b model. The size of the vocab with 1/2/3 gram forces the Engram table table to a full 1b. That's 50% where

More Gen AI coverage and frameworks integrations to minimize code changes New models supported: On CPU: Gemma-3n On CPU, GPU: Kokoro-82M, Qwen3-VL-4B with eagle3, Qwen3-ASR, Muse Glimmer 30B, Qwen 3.8 27B, Gemma4 12B; Hy-MT2-1.8B, DeepSeek OCR-2, Granite 4.0 H Micro On NPUs: FLUX.2-Klein 4B, Kokoro 82M Additional CPU and GPU-enabled models available as early releases: Qwen-image, Z-Image-Turbo, Granite 4.0 H Tiny, Fun-ASR-Nano, LFM2.5-8B-A1B, MiniCPM5-2B, RF-DETR, BGE Reranker-V2-M3, BGE M3 Broader LLM model support and more model compression techniques OpenVINO™ GenAI adds Multi-Token Prediction (MTP) speculative decoding for Gemma 4, Qwen 3.5, and Qwen 3.6, on CPUs & GPUs enabling higher throughput and lower latency without sacrificing accuracy. With Tree Drafting (Top-K) for EAGLE3 now

Hey Reddit. This is a post asking for advice / user experience. The goal is simple: deploy a small private server for a developer to run a harness that rivals/beats Claude Opus (in perf/intelligence, not necessarily speed). I believe the model to target is a Q3 or so version of the Qwen 3.8 Next model, and to build a dual GPU, 64GB setup. Streaming n-gram off RAM/SSD (~85GB for total model + KV, minus 27 n-gram = 63GB) would mean the 64GB might be enough. Budget: as low as possible, up to $4000. I am willing to get extremely scrappy though. I am wondering: if I want to run this model with a full 256k context, and want about ~500 tok/sec PP, and +20 tok/sec decode, what kind of cards do I need? I see my options as follows: Budget and weird Something super silly with 2 or 3 Tesla V100s, the

I find another new model at HF: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B "Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat ), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks. For more information, please refer to our GitHub repository . Highlights Agent-Oriented Architecture : Built on the mHC + MLA + MTP architecture, supporting multi-step planning, tool calling, and complex reasoning chain execution, ensuring task coherence and

Everyone now talks about the architecture that's not auto regressive and does lightning fast probability prediction with a json schema. I worked on this literally one year back in March 2025, published an arxiv paper, pushed the model to huggingface along with the pypi package and training dataset. And then one year later, a frontier lab came, proposing the same idea like literal breakthrough without technical papers, open weights and no open dataset. I posted my approach in this subreddit. Links are. For anyones information the main guiding model is RL not embedding model or LLM Reddit post: https://www.reddit.com/r/LocalLLaMA/s/6eGEwsAz43 Paper: https://arxiv.org/abs/2503.23303 Model: https://huggingface.co/DeepMostInnovations/sales-conversion-model-reinf-learning Dataset: https://huggin

Everyone now talks about the architecture that's not auto regressive and does lightning fast probability prediction with a json schema. I worked on this literally one year back in March 2025, published an arxiv paper, pushed the model to huggingface along with the pypi package and training dataset. And then one year later, a frontier lab came, proposing the same idea like literal breakthrough without technical papers, open weights and no open dataset. I posted my approach in this subreddit. Links are. For anyones information the main guiding model is RL not embedding model or LLM Reddit post: https://www.reddit.com/r/LocalLLaMA/s/6eGEwsAz43 Paper: https://arxiv.org/abs/2503.23303 Model: https://huggingface.co/DeepMostInnovations/sales-conversion-model-reinf-learning Dataset: https://huggin

I've been working on something that started as the training backend for my Hierarchos architecture, but it has grown into a much broader project: https://github.com/necat101/Hierarchos-Native Hierarchos Native now includes a native Rust + Vulkan backend for training and inference across 143 canonical Transformer architectures , with additional Hugging Face model_type aliases bringing the currently advertised registry to 224 model-type spellings . The part I'm most interested in sharing isn't actually Hierarchos itself anymore. It's the Transformer backend underneath it. The goal is to explore how much of the modern Transformer ecosystem can be implemented directly on Vulkan compute rather than assuming CUDA + PyTorch as the default execution stack. The backend currently includes native imp

I let Qwen 3.8 27B 4bit quantized with 100K context window run autonomously for 63 hours (50 million+ tokens) to try to solve the RH. Of course it did not solve it, but the experiment still shows it's internal work, memory organization, strategies used and more. The interesting thing is that it never hallucinated an answer and never stopped trying new ideas to solve it. Multiple times it corrected it's own mistakes. I am really hopeful that one of the unsolved millenium prize problems will be solved by an agent or a swarm of agents powered by an open source model in the next 12 months. If you want to check out it's internal memories, code, strategies and more I published everything on HF: https://huggingface.co/datasets/gr0010/artificium-riemannhypothesis-experiment My next goal is to actu

Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and Grok 3 were already enough for general chatting,search lookup, RP, etc. So if you brought those models back today, I feel like a lot of ordinary users wouldn't notice a massive difference for " EVERYDAY GENERAL USE" So how far has the sub-10B SOTA actually come? For example, MiniCPM5-2B is only ~2.5B parameters but has 131K context and strong current results in math, coding, long-context, tool use and agentic tasks. If you combine something like MiniCPM5-2B (including an abliterated/heretic variant) + modern harness + tool calling + web search + RAG/external memory + Python/code execution + fil

With the recent Nvidia Huggingface acquisition and frontier AI labs screaming about safety and putting guardrails everywhere, I think it's important that we have local models that aren't affected by arbitrary guardrails set during training. To be clear, this post NOT about Enterprise Resource Planning (ERP). Censorship in an LLM can highly affect its abilities to do many legitimately useful things (note GPT-OSS, Fable 5), and going forth I believe censorship will only get more and more strict. There have been many resources and posts about uncensored models using abliteration, heretic, and probably many other methods that I'm not aware of. However, it seems like all of this information is scattered about the place, and Huggingface is essentially flooded with "uncensored" variants of basica

Apple is considering offering an AI server built around "M8" series chips and has discussed incorporating Nvidia networking hardware, The Information reports. The system would be sold to outside customers, potentially bringing Apple back into a business it left behind when it discontinued Xserve in 2011. Apple is reportedly targeting companies that want to run AI models on their own equipment, with a particular focus on inference and generating responses from trained models. One option under consideration is Nvidia's NVLink Fusion, which provides hardware and software for exchanging data between chips. Apple could use the platform to link its M8 processors, although neither the Nvidia arrangement nor the server itself is finalized. The report places a potential release date in 2029, but it

We gave two small LLMs a real customer-service task. One got 100/100. The other stopped with the job half-done. I wanted to test something more meaningful than benchmark scores: can a small model actually complete a multi-step agent task without handing the work back to the human? The task: A customer says: “My order is MB20260908017. The left earbud has no sound. I'd like a replacement. Could someone pick this one up tomorrow afternoon?” The agent has to: Look up the order Check the replacement policy Check nearby inventory Create the replacement Schedule the pickup Reply with the actual reference numbers The catch: the closest warehouse is Shanghai, but it has 0 stock . Suzhou is 95 km away with 12 units. Also, the system date is Sept 9 , so “tomorrow afternoon” means Sept 10 , not some

Recurrent Looped Transformer (RLT) passes the decoder's final hidden state to the next token, together with that token's causal encoder representation. The decoder reads encoder-derived global KV memory and maintains a sliding-window attention (SWA) cache at every layer. The same update runs over prompt and response tokens. More effective reasoning depth! https://github.com/yifanzhang-pro/recurrent-looped-tranformer submitted by /u/WebAssemblyMan [link] [comments]

https://preview.redd.it/e5wn8eyh7vph1.png?width=1080&format=png&auto=webp&s=5faeec866acc9e8eca8dc9e84a6661bee7bcf954 https://preview.redd.it/8ov5gl8j7vph1.png?width=1080&format=png&auto=webp&s=440320ca4646ff7dceaf2aa5ac2ef76faeeba525 New Qwen3.8-Flash-Next quantization using GSQ-RCO. Cuts the size of Qwen3.8 Flash Next from around 80-95GB to 68-76GB, while still preserving near baseline quality. Also their Q2_0 variant claims to be much faster offering 6.2x better prompt throughput in coding. "Q2_0 is built for speed. It avoids the quantization formats that rely on large lookup tables: those formats pack more accuracy into a given bit-width, but decoding them costs real time, and on this model that cost dominates inference. Q2_0 delivers 3.4x the prompt throughput and 1.9x lower end-to-end

A follow-up to my earlier post about small models on phones. Last time it was fixed tasks in a single browser. This is something else. Ministral 3 3B Q4_K_M in llama.cpp on a Samsung Galaxy S21, four CPU threads, no GPU, running on battery, relaying between Chrome with Gemini open and Firefox with Z.ai open. There is no direct integration between the two services. The model on the phone works with both web interfaces, and the script carries the replies between them. After a round of testing we did 10 runs as a series and got 10 out of 10: four browser hops each, forty replies, twenty automatic resets to a new chat. Run time from 7:59 to 11:55, median 9:29. Over the 98 minutes of the series the battery went from 79% to 63%. What I find interesting is this: The script contains no per-site co

I work with one of Andrew NG's companies around document space. Our document pipeline hits three vendors for one flow: Textract for OCR, OpenAI for embeddings, Cohere for reranking. Three bills, three SDKs, three sets of rate limits, and the data leaves our cloud three times. Trying to work out whether one endpoint can do the whole thing. The managed vendors don't consolidate, so it means self-hosting. What I've found: Piece it together: TEI plus a separate reranker plus docling for OCR. Works, but that's three deployments, which is the thing you were escaping Infinity: covers embeddings and reranking from one server, but not OCR, so Textract still needs its own replacement SIE: the only option I've found that does all three behind one API. Superlinked lists docling at $0.397 per 1,000 pag

It's late at night in the lab, I've been working on a basic script for a virology project, and holy hell the safeguards have been pissing me off. Mirroring detectEVE data over rsync to my laptop by making a zip file first? No no no, great safety mogul DARIO demands there be NO file transfer today. Request blocked, reported, labeled [cyber]. Yet, Deepseek V4.1 does it with no complaint. I got tired of reading papers - so I ask Claude - "Does this PDF go over binary host virus infections?" Immediately blocked for biological safety risk. Deepseek V4.1 tells me it doesn't have the data I need without drama. Bioinformatics server goes down and I need help getting it back up by getting the outputs of my diagnostic scripts to the mounted usb drive? Oops, its named exfil. Looks scawy. No transfer