Jul 24th, 2026

Let’s be honest: I have a bit of a love-hate relationship with Google’s new AI Mode.

On one hand, it’s free. On the other hand, it’s a black box. I use it constantly for those “hey, how does this tech thing work?” questions that don’t involve my bank password or my pet’s medical history (eh, I don’t actually have a pet, just sayin’). It’s powered by Gemini 3.5 Flash, a model that Google keeps under heavy lock and key. They won’t tell us exactly how the sausage is made, but independent engineers have done some serious detective work on that and made it searchable public data.

Here’s the tea: Gemini 3.5 Flash is an ultra-sparse Mixture-of-Experts (MoE) model. It’s got a massive brain—somewhere between 250 billion and 300 billion parameters total. But here’s the magic trick: it doesn’t use all of them at once. For any single request, it only wakes up about 10 to 16 billion parameters. This “sparse” approach lets it run about four times faster than a dense model without losing its mind. It’s efficient, it’s fast, and it’s proprietary.

But since I like to tinker, I wanted to see if I could replicate that magic locally, without sending my data to Mountain View. So, I ‘asked’ Google AI for an open-source alternative. It gave me Gemma 4.

The Challenger: Gemma 4 31B

Gemma 4 31B – It’s a dense model from Google (yes, the same company, but open-source this time). According to BenchLM.ai, and the results were surprisingly close.

In fact, Gemma 4 31B beat Gemini 3.5 Flash in general benchmarks, scoring a 52.9 compared to Flash’s 47.2. It’s a beast for writing essays, following strict instructions, and parsing complex data. If you need an AI that writes clean JSON or builds a simple app from a single prompt, Gemma is your friend. It’s smart, it’s conversational, and it’s great for human-like chat.

But here’s the catch: it’s dense. That means every time it thinks, it uses all 31 billion parameters. It’s like trying to drive a semi-truck through a neighborhood. It gets the job done, but it’s heavy, slow, and burns a lot of fuel (or in this case, GPU VRAM).

The Heavyweight: Qwen 3.6 35B-A3B

Then there’s Qwen 3.6 35B-A3B from Alibaba. This one is also an MoE model, but it’s built differently. It has 35 billion total parameters, but like Gemini, it only activates a tiny fraction of them—just 3 billion—per token.

At first glance, Gemma and Qwen seem like rivals. They’re both mid-sized, open-weight, and capable. But they excel in completely different arenas.

Gemma 4 31B wins when:

• You want the most natural, human-like conversation.
• You need strict adherence to formatting (like scaling bounding boxes into JSON).
• You’re writing a one-shot script or essay.
• You’re testing broad knowledge and reasoning.

Qwen 3.6 35B-A3B wins when:

• Speed matters. Because it only uses 3B active parameters, it’s blisteringly fast. We’re talking up to 4x higher token throughput than Gemma.
• You’re building agents. Qwen dominates in multi-step tool use and terminal interactions.
• You’re dealing with massive context. Qwen can handle huge streams of images or video frames that would choke Gemma’s processing limits.
• You’re debugging large codebases. Qwen is optimized for navigating complex, existing repositories, not just writing snippets from scratch.

Why Qwen is the Brain for Hermes

So, if you’re building a local AI agent—specifically, the Hermes AI Agent—which one should you pick?

If you want a chatbot to write poetry, pick Gemma. But if you want an autonomous agent that can think, act, and react in real-time, between these two, there is one clear winner: Qwen 3.6 35B-A3B.

Here’s why Gemma fails as an agent brain, and why Qwen succeeds:

1. The “Agent Loop” Problem

Agents don’t just chat. They run hidden loops: Reason → Tool Call → Execute → Observe → Output. This happens over and over again, often in milliseconds.

• Gemma crawls at about 25–35 tokens per second. By the time it finishes one step of the loop, you’ve already checked your email twice.
• Qwen generates at 110–145 tokens per second. It’s fast enough to feel instant. A slow agent feels broken; a fast agent feels like magic.

2. Function Calling: Trust Issues

An agent needs to call tools (like opening a terminal or searching the web) without hallucinating or breaking character. Qwen is explicitly trained on native tool-use and JSON function calling. It dominates benchmarks like Tau2, meaning it actually triggers your Hermes Agent’s tools reliably. Gemma? It’s smart, but it can get confused by nested agent prompts and might start making up arguments instead of executing commands.

3. Memory Management

Agents accumulate a lot of “dirty” history—terminal outputs, system messages, error logs. Qwen handles this messy, multi-turn context incredibly well. Gemma, being dense and rigid, tends to get overwhelmed by the sheer volume of agent-specific noise.

The Bottom Line

Gemma 4 31B is a phenomenal model for content creation and structured tasks. But for autonomous agency, speed and reliability are everything.

Qwen 3.6 35B-A3B gives you the speed of an MoE model with the reasoning power to handle complex workflows. It’s the only choice that keeps your Hermes Agent feeling responsive, reliable, and actually useful.

So, keep Gemini for your casual chats. But for your local agent? Fire up Qwen. Your GPU (and your patience) will thank you.

** That said, I am still testing out Qwen with Hermes, its still hasn’t satisfy my own expectations but local AI will still be moving progressively forward by leaps and bounds, so I think we will see very usable cases for local AI more sooner than we think in the very near future.