The Ultimate Local AI Coding Setup: VS Code + Continue + Gemma 4 (No GPU Needed)

Want an AI coding assistant that runs entirely on your machine, costs nothing to use, and doesn’t send a single line of code to the cloud? With VS Code, the Continue extension, Ollama, and Google’s Gemma 4, you can build a powerful local coding agent that genuinely feels like a replacement for GitHub Copilot—without needing a GPU.

This guide walks through the full setup from scratch, including which Gemma 4 model to pick for your hardware, how to configure Ollama, and how to turn Continue into a fully autonomous coding agent inside VS Code.

How the Local AI Coding Stack Fits Together

This setup has four main pieces that work together to turn your laptop into an AI-powered coding machine:

1. VS Code – Your main code editor. Nothing special here; it’s just the environment where you write and manage your code.

2. Continue – An open-source AI coding agent extension for VS Code. It’s free, Apache 2.0 licensed, and a serious alternative to GitHub Copilot.

Continue has two important modes:

Chat mode: Simple Q&A. You ask questions, it replies, but it doesn’t act on your files.

Agent mode: This is where it becomes powerful. In agent mode, Continue can:

• Read and write files
• Edit existing code
• Run terminal commands
• Use tools like file search and diffing

It can do all of this autonomously, without asking for your approval on every step if you choose.

3. Ollama – A local model server that runs on localhost:11434. Continue sends HTTP POST requests to Ollama with your prompts, and Ollama forwards them to the model and streams back responses.

4. Gemma 4 – Google’s open-source model that actually generates the code and reasoning. It’s loaded into RAM by Ollama and never leaves your machine.

The flow looks like this:

You type a request in Continue → Continue sends it to Ollama on localhost:11434 → Ollama passes it to Gemma 4 → Gemma 4 generates a response → Continue uses that response to create/edit files or run commands in your workspace. All of this happens locally, with zero network traffic.

Choosing the Right Gemma 4 Model for Your Hardware

Before installing anything, it’s worth picking the right Gemma 4 variant for your machine. Head to the Ollama model library and look up Gemma 4. You’ll see it supports vision, tool use, audio, and more—but for coding, the key is which size and architecture you choose.

Edge Models: For Lighter Hardware

These are optimized for modest machines:

Gemma 4 E2B / E4B (Edge models)

• E = Effective parameters (only a subset of total parameters are active at a time)
• E4B has 8B total parameters, but only 4B active during inference
• Around 9.6 GB download
• 128K context length
• Quantized (Q4_K_M) and optimized for local use

If you have under 16 GB of RAM, this is your default choice. It’s lighter, faster, and still very capable for everyday coding tasks.

Workstation Models: For Heavier Reasoning

If you have more memory and want stronger reasoning, look at the workstation models:

Gemma 4 26B (Mixture of Experts)

• 26B total parameters, but only ~4B active at a time (Mixture of Experts)
• You get the reasoning quality of a large model at the compute cost of a smaller one
• ~18 GB download
• 256K context length

This is a great option if you have around 24 GB of RAM and a decent CPU. It’s a sweet spot between capability and performance.

Gemma 4 31B (Dense)

• 31B dense parameters (all fire on every pass)
• More consistent outputs, but heavier and slower
• Best suited for 32 GB+ RAM

One important detail: Mixture of Experts (MoE) saves compute, not RAM. You still need enough memory to load all the weights, even though only a subset of parameters is active at a time.

For the build described here, the 26B MoE model is used on a Windows machine with 32 GB of RAM, but the same approach works with smaller models if your hardware is lighter. If you want a more general walkthrough of this kind of setup, you may also find this Gemma 4 + Ollama + VS Code guide useful.

Installing Ollama and Running Gemma 4 Locally

Once you’ve chosen a model, it’s time to install Ollama and pull Gemma 4.

1. Install and Verify Ollama

On Windows: open PowerShell and run the one-line install command from Ollama’s website. It downloads and runs the installer for you—no manual EXE downloads or wizards.

After installation:

• Run ollama --version in PowerShell to confirm it’s installed.
• Open a browser and go to http://localhost:11434. You should see a simple status page saying Ollama is running.

2. Pull the Gemma 4 Model

In PowerShell, run the pull command for your chosen model, for example:

ollama pull gemma4:26b

This will download roughly 17–18 GB for the 26B model, so it may take a while. You’ll see the manifest and model layers downloading one by one. When it reaches 100% and shows success, Ollama will drop you into a REPL for that model.

Test it with a simple coding prompt, like:

Write a Python function to reverse a linked list.

You should see a detailed answer with iterative and recursive solutions. While it’s generating, you can open Task Manager and confirm:

• CPU usage is high for ollama
• Memory usage is around 18–19 GB for the 26B model
• Network usage is effectively zero

That’s your proof it’s running fully offline.

Type /bye (or the equivalent exit command) to leave the REPL.

3. Increase the Model Context Length

By default, Ollama may load Gemma 4 with a context length of 4096 tokens. That’s fine for simple chat, but it’s not enough for an autonomous coding agent that needs to:

• Read multiple files
• Include tool outputs
• Keep conversation history
• Handle long prompts and instructions

Once the context fills up, the model silently forgets earlier parts of the conversation, which can break complex coding flows.

Ollama doesn’t expose a simple CLI flag to change context length per run. Instead, you define a custom model variant using a Modelfile (sometimes referred to as a model file) and bake the context length into it.

For example, you can create a variant called gemma4-agent with a 16,384-token context. The Modelfile only needs a couple of lines to:

• Base it on the original Gemma 4 26B model
• Set num_ctx (or equivalent) to 16384

After building and running this variant, check the model info in Ollama. You’ll see:

• Model: gemma4-agent
• Context: 16384
• Size in RAM: still ~19 GB

Context length doesn’t change the base model weight size; it just affects the KV cache on top. That means you get 4x more context headroom without increasing the core memory footprint.

Connecting Gemma 4 to VS Code with Continue

With Ollama and Gemma 4 ready, the next step is to wire everything into VS Code using the Continue extension.

1. Install the Continue Extension

In VS Code:

• Open the Extensions panel (Ctrl+Shift+X)
• Search for Continue
• Install “Continue – Open-source AI coding agent”

After installation, restart VS Code if needed. You’ll see a new Continue icon in the sidebar. Click it to open a chat-like panel.

2. Add Ollama as a Model Provider

At the bottom of the Continue panel, click the model selector and choose to add a chat model. In the provider dropdown, you’ll see options like Azure OpenAI, Google Gemini, Mistral, Ollama, and OpenAI.

Select Ollama as the provider.

For the model, you can either:

• Use auto-detect, which lets Continue pick up whichever model Ollama has loaded, or
• Hardcode a specific model name (recommended for stability)

To start, choose auto-detect and click Connect. Continue will generate a configuration.yml file and add an entry for Ollama.

3. Clean Up and Hardcode Your Gemma 4 Variant

Open the generated configuration.yml. You might see multiple model providers listed (for example, leftover LM Studio entries from previous setups). To avoid confusion:

• Remove any providers you’re not using (like LM Studio).
• Replace the auto-detect Ollama entry with a specific model name, such as gemma4-agent, your custom 16K-context variant.

Save the file. Back in the Continue panel, open the model dropdown again—you should now see only your Gemma 4 agent model listed. This guarantees that Continue always talks to the exact model you tuned for coding.

Turning Continue into a Fully Autonomous Coding Agent

With the model connected, it’s time to use Continue as a real coding agent, not just a chat assistant.

1. Understand Tools and Modes

Continue comes with a set of built-in tools, such as:

• Read file
• Create new file
• Edit existing file
• Run terminal command
• View differences
• File glob search

Each tool has a behavior setting:

• Automatic – The agent can use this tool without asking you.
• Ask first – The agent pauses and requests your approval before using it.

By default, safer tools like “read file” may be automatic, while more sensitive ones like “create new file” and “run terminal command” are set to “ask first.”

However, there’s a key detail: in Chat mode, tools are disabled. That’s why, if you ask it to create a file in chat mode, it will only show you code with a “Create file” button instead of writing directly to disk.

To unlock the full power of tools, you need to switch to Agent (or Plan/Agent) mode.

2. Switch to Agent Mode

At the bottom of the Continue panel, change the mode from Chat to Agent.

Now, when you give it a task, Continue can:

• Inspect your project structure
• Create and edit files
• Run terminal commands
• Fix its own mistakes (for example, renaming a file via mv if it misnamed it)

You can also open the tools settings (gear icon → Tools) and choose which actions should be automatic vs. ask-first, depending on how much control you want.

3. Give It a Real Project: Building a Landing Page

To see the agent in action, create a new empty folder for your project and open it in VS Code. With Continue in Agent mode and your gemma4-agent model selected, try a more complex prompt, such as:

“Build a complete landing page for a startup called Pitch Zero, an AI-powered pitch scoring platform. Include:

• A hero section with headline and CTA
• A feature section
• A ‘How it works’ section
• Pricing tiers
• A dark, professional theme with purple accents
• Three separate files: index.html, style.css, and app.js.”

In Agent mode, you’ll see a very different behavior compared to simple chat:

• It first explores the current directory using the read tools.
• Then it creates index.html and opens it in a new tab.
• Next it creates style.css and app.js automatically.
• If it makes a mistake (for example, naming the CSS file index.css instead of style.css), it can detect that, run a terminal command like mv index.css style.css, and continue without you intervening.

Within minutes, you’ll have:

• A full hero section with CTAs
• Feature cards (e.g., instant scoring, actionable insights, investor readiness)
• A “How it works” section with step-by-step explanations
• Pricing tiers (Starter, Pro, Enterprise)
• A dark, professional layout with purple accents
• Smooth scroll animations and a responsive layout handled in app.js and style.css

Open index.html in your browser (for example, via a simple PowerShell command or by using a live server extension) and you’ll see a complete, polished landing page generated from a single prompt—entirely offline.

Why This Local Setup Matters

This kind of stack—VS Code + Continue + Ollama + Gemma 4—shows how far local AI coding has come:

• Zero cloud dependency: Your code never leaves your machine, which is ideal for sensitive or proprietary projects.

• No API keys or usage limits: Once the model is downloaded, you can use it as much as you want.

• Competitive with cloud tools: For many coding tasks, a well-configured Gemma 4 model can feel similar to GitHub Copilot, Cursor, or other cloud-based code assistants.

• Hardware-friendly: You don’t need a GPU—just a decent CPU and enough RAM for the model size you choose.

If you’re exploring other free coding setups, especially around Claude, you might also like this guide on running Claude Code for free with Ollama and Gemma 4.

Once you’ve gone through these steps, you’ll have a fully functional, private, and powerful AI coding agent living inside VS Code—ready to help you ship projects faster, without giving up control of your code.