For about a week every winter I pretend I'm going to make video games a sustained hobby. In the blink of an eye, my employer's shutdown ends, customer needs spin back up, and I'm left with some current-gen GPU collecting dust... until now.
Ever since my hard pivot to Linux this year, I've wanted to test the waters on building a local LLM rig. If you combine a moderately powerful GPU (w/ 16GB of VRAM), the release of Gemma 4 (12B specifically), and some free time, you wind up with a pretty rad little local setup using Gemma 4, Open WebUI, and VS Code.
Here's Gemma helping me setup a local server with a self-signed cert:
To appreciate this, you have to think about how our industry has changed that last two years. Most of my dev work has pivoted to being mostly agentic-driven development (i.e. tool calling in a loop). Unless I'm doing very specific UI and UX work, I almost never write code now.
The elephant in the room here is that we're all using Claude in the cloud to achieve this. You ask Claude a question, it gets sent up to Anthropic's servers, and you receive a response back. This is expensive by every definition of the word: big data centers chew through energy, I'm paying Anthropic a cool $100 / month for the privledge, etc. etc. Up until recently, tool calling with local LLMs either didn't exist or was bolted on after the fact.
Gemma 4 (including 12B) has native, token-level tool calling with typed argument encoding. In contrast, Gemma 3 relied on prompt-engineered JSON. Other open-weight models like Qwen 2.5 sat in between — it was trained-in and reliable, but used text-JSON inside elimiters rather than custom typed encoding.
Now, is Gemma as good as Claude? Hell no. 31B might be maybe 60-80% of Opus on paper for similar tasks, but practically speaking it's much lower due to the context window size and the ecosystem around Claude. Tools, skills, user experience are all still much more polished.
I'm fascinated by all of this in two ways. First: I'm fascinated by the progress. I am running something locally that has roughly the same quality of what frontier models were doing two years ago. It wasn't that long ago we had our little chat pane up in Cursor working on one file at a time. That's effectively where a local model is reliably today (with a bit of modern affordances like tool calling). The next fascination I have is how much these developments parallel all of computing history. Technology shrinks and is made cheaper and more accessible over time. We used to have mainframes and punch cards and now we have smartphones. Seeing AI shrink like this makes me really excited about the future.