Completed my operational setup of a private Local AI Chat that can process audio, video, images, text file, PDFs and can even hook up with MCP servers. And my choice of AI model? I chose the Qwen 3.6 35B-A3B UD Q4_K_M MTP model finetuned by unsloth studio.

The 35B-A3B Architecture is an MoE (Mixture of Experts) system with 35 billion total parameters, but only 3 billion active parameters per token. You get the deep knowledge base of a 35B model with the blistering processing speed of a tiny 3B model.

With UD Q4_K_M, its Unified / Dynamic imatrix quantization. This means it has been heavily optimized for LLAMA.CPP down to a ~4-bit file size (~22.7 GB) with minimal loss in reasoning accuracy.

The MTP (Multi-Token Prediction) allows the model to predict multiple tokens at once. It acts as a built-in speculative decoding engine, driving generation speeds up by 1.4x to 2.2x (reaching up to 240 tokens/sec on specialized GPUs) without sacrificing accuracy.

How It Compares to the Best AI Models of 2026?
When it comes to multi-step execution—like scanning code repositories, terminal commands, or using Model Context Protocol (MCP) integrations—this model hits far above its weight class.

On the specialized MCPMark tool-use benchmark, Qwen 3.6 35B-A3B scores 37.0%, more than doubling the 18.1% achieved by Google’s popular Gemma 4-31B.

The MoE Edge: Because it only activates 3B parameters, an agent can run continuous, iterative terminal tasks locally without choking on hardware or driving up massive cloud API bills.

Local vs. Frontier Closed Models – The Gap: If you compare it to cloud titans like GPT-5.5 (~88.7% SWE-Bench) or Claude Opus 4.8, Qwen 3.6 35B-A3B sits slightly behind on raw logic, posting a respectable 73.4% on SWE-Bench Verified.

The Victory: For a local, quantized model that runs completely offline on consumer hardware, a 73.4% score is unprecedented. In niche community tests, its native vision encoder and reasoning “thinking mode” have even out-rendered older premium models on multi-step visual prompts.

Hardware Requirements & Speeds: Unlike the dense models of 2026 that require massive server racks, this specific Q4_K_M file fits tightly into local consumer rigs. It runs smoothly on high-end consumer desktops or Macs with 24GB to 32GB of unified memory/VRAM.Thanks to the MTP spec-decoding integration, it produces local text at speeds that make dense 30B+ models feel sluggish.

My AI h/w setup is a ROG Asus z490-F, 49GB DDR4 RAM, 2x Nvidia RTX2080Ti c/w 22GB each, 1000W PSU. Will try a NVLink on the twin GPUs when it arrives. Tried to do RDP (Remmina) but failed miserably due to buggy issues between Ubuntu Wayland and X11 Linux Mint, but thankfully, resolved it with Tailscale and RustDesk Client setup that makes it real fast and secure. Saved so much time on setup headaches with these two solutions combined.

Onward to setting up Hermes AI agents next.

Updated Sep 21st, 2026:- Running on Qwen 3.8 Flash Next Q2 on this same hardware setup. The main 37Gb model sits in VRAM in split-tensor mode and the 28Gb N-gram embedded file sits in fast NVME. Performance via local IP on llama.cpp is still amazing at 316 tokens, 7.2s at 44.16 tokens/sec, but I noted that for Hermes AI agent, it trudges along at almost half the speed on average, probably due to my internal homelab routing traffic (my Hermes LXC is sitting in another server elsewhere).