Local LLMs for fun

Published 2026-07-05

Yeah, programming with Large Language Models (LLMs) has gotten quite good. One year ago I was quite sceptical of how well they could perform in real development, but with a combination of improvements in agentic LLMs and, to be honest, my skills with wielding them, it would be quite difficult to put the genie back into the bottle. However, as the third-world governments keep playing oddballs with limiting access to frontier models and we really can’t keep relying on the goodwill of multibillion dollar companies for our tools, it seems reasonable to assume that this isn’t going to last forever. While it probably would be pragmatic to just use something like OpenRouter to use smaller and more open model providers for quite cheap, thats no fun at all compared to trying to get worse results by spending more time and paying more.

As I happen to have a decent enough GPU in my desktop (RTX 3060) with 12 GB of VRAM and 32 GB of normal RAM, I wanted to see if I could actually run something useful with just local models. The quite recently released Qwen3.6-35B-A3B had been getting quite a bit of attention for a great size to performance ratio, and it seemed that with sensible quantizations it could fit the active parameters of its mixture of experts easily into my VRAM, while leaving quite a nice amount left over for a large context.

For serving the model, Llama.cpp seemed to be a good option for playing around with a bunch of options to try and get as much juice out of my hardware as possible. Interfacing with the model would be done with OpenCode for a simple, out-of-the-box experience.

Initially, I tried the Unsloth Qwen3.6-35B-A3B-UD-Q4_K_M.gguf model just to see how everything works. Running llama-server with the initial command of:

llama-server \
    --model ./model/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    --n-gpu-layers 99 \
    --n-cpu-moe 99 \
    --ctx-size 16384 \
    --flash-attn on \
    --host 127.0.0.1 \
    --port 8080 \
    --alias qwen3-a3b

and the OpenCode config of:

{
    "$schema": "https://opencode.ai/config.json",
    "provider": {
        "llamacpp": {
            "npm": "@ai-sdk/openai-compatible",
            "name": "llama.cpp (host)",
            "options": {
                "baseURL": "http://host.docker.internal:8080/v1",
                "apiKey": "dummy"
            },
            "models": {
                "qwen3-a3b": {
                    "name": "Qwen3 35B A3B (local)"
                }
            }
        }
    }
}

I got it working, and was quite surprised at how well it could generate code with just a vague prompt, such as

i want to have a project that gets all the news from this page: https://kiipeilyareena.com/ajankohtaista/ and then creates an rss feed from it. i want it to be a python project using uv, and it should regenerate the rss feed hourly on github pages using github actions.

it eventually managed to get me a working project with a repo at jonesus/kiipeilyareena-rss, although I gave it a few more steering prompts during the run (discussion transcript readable at https://opncd.ai/share/w4GOyQ1n).

During the experimenting, I quantized the KV cache to --cache-type-k q8_0 --cache-type-v q8_0, increased the context to --ctx-size 131072, and offloaded only --n-cpu-moe 30 experts to RAM. This gave me a nice bump in performance, and during OpenCode use, I saw prompt processing performance of ~200 tokens per second and encoding performance of ~20 tokens per second (as observed from the llama-server logs).

I still thought that there must be some way to make this more performant; after finding this Reddit thread on someone else’s optimizations with similar hardware, I wanted to try the same approach myself. Instead of the mainline inference server, they used a llama.cpp fork by spiritbuun, and instead of Unsloth’s quantized model weights, they used mudler’s APEX quantized weights. The fork allows us to run TurboQuant for the KV cache, so that we can get much larger context for the same VRAM, and the APEX weights are a bit more optimized for selective quantizations, leading to smaller file size and thus less MOE offloading to the cpu. For my use cases, I have found a long context to be invaluable, so that the agents can iterate on for a long time using external tools, but who doesn’t also love speed?

From here, I iterated on finding an optimal tradeoff between speed, KV quantization and context length. I don’t really love relying on forks of well-maintained projects as they tend to diverge and wither, so I wanted to see how much there was really to be gained by using the fork and spinoff weights. I went for sweeping a large amount of permutations for easily quantifiable measurements like tokens per second (via llama-bench), perplexity (via llama-perplexity), and a dirty --n-cpu-moe vs. --ctx-size vs VRAM usage script by Claude.

Testing again with some throwaway coding requests, I was now seeing prompt processing performance of almost 500 tokens per second, and encoding performance of ~30 tokens per second. With a simple needle-in-the-haystack test, the 262k context also seemed to work deecntly well. The final command for launching the server ended up to be:

buun-llama-server \
    --model ./model/Qwen3.6-35B-A3B-APEX-I-Compact.gguf \
    --n-gpu-layers 99 \
    --n-cpu-moe 28 \
    --ctx-size 262144 \
    --flash-attn on \
    --cache-type-k turbo4 \
    --cache-type-v turbo4 \
    --no-mmap \
    --parallel 1 \
    --host 127.0.0.1 \
    --port 8080 \
    --alias qwen3-a3b

From here, I’ll have to find some time later on for next steps:

1. Try to see if this is actually useful, not just for simple throwaway scripts, and
2. Work on the harness; https://pi.dev/ seems quite promising.