Hey everyone, Alex here from Coding with Alex. If you’ve spent any time on Hacker News, Reddit, or your team's Slack channels lately, you’ve probably noticed a massive, shifting tide in how we think about developer productivity tools. For the last two years, we’ve been told that the future of coding is hosted AI: sending our proprietary codebases, secret API keys, and intellectual property up to third-party APIs in exchange for autocomplete suggestions and boilerplate generation.
But lately, things are changing. The conversation is shifting away from massive, multi-billion-dollar cloud APIs toward something much closer to home: local-first development tools. Whether it's due to strict corporate compliance, flight cabin offline modes, latency issues, or just a fundamental desire to keep your IP on your own silicon, developers are demanding high-performance, private, and offline-capable AI workflows.
Today, we are going to roll up our sleeves and build exactly that. We’re going to bypass the subscription paywalls and telemetry trackers to set up a blazing-fast, fully local AI coding assistant inside two of the most popular editors today: Neovim and VS Code. We will configure a local inference engine using Ollama, pull down state-of-the-art coding models, and integrate them directly into our editor workflows with zero external network requests.
Why Local-First AI Matters for Developers
Before we look at the configuration, let’s talk about the why. Why should you care about running local models when GitHub Copilot or Claude 3.5 Sonnet are just an API call away?
- Zero Latency (Once Cached): Round-tripping to a server in Oregon or Dublin takes time. When you are editing code, even a 300ms delay in autocomplete suggestions ruins your flow state. Local models running on Apple Silicon (M1/M2/M3/M4) or modern NVIDIA GPUs can achieve token generation speeds that feel instantaneous.
- Security and Compliance: If you work in fintech, healthcare, or defense, pasting code into a cloud-hosted LLM is a fireable offense. Local models run entirely in your local RAM and VRAM. Your code never leaves your machine.
- Offline Capability: Whether you are on a train, an airplane, or dealing with spotty conference Wi-Fi, your development environment remains 100% functional.
- Cost Predictability: No seat licenses, no usage tiers, and no surprise API bills at the end of the month. You pay with your hardware and your electricity bill.
The Architecture: How Local AI Inference Works on Your Machine
To make this work, we need three distinct layers in our local stack:
- The Weights (The Model): The trained neural network. For coding, we want models optimized for Fill-In-The-Middle (FIM) tasks, like
deepseek-coder:6.7b,qwen2.5-coder:7b, orcodellama:7b. - The Inference Engine (The Server): A local background service that loads the model into your system memory/VRAM and exposes a standardized API (usually mimicking the OpenAI chat completions schema). We will use Ollama for this because of its simplicity and excellent resource management.
- The Editor Plugin (The Client): The bridge inside your editor that captures your cursor position, drafts a prompt, sends it to your local API, and streams the output back to your active buffer.
Here is a simple conceptual map of how data flows in our local setup:
+--------------------------------------------------------+
| Your Machine |
| |
| +------------------+ +---------------------+ |
| | Text Editor | | Inference Engine | |
| | (VS Code/Neovim) | | (Ollama) | |
| | | | | |
| | [Cursor Context] -------> | [Port 11434 /API] | |
| | [Inline Suggest] <------- | [Streamed Tokens] | |
| +------------------+ +----------+----------+ |
| | |
| v |
| +---------------------+ |
| | Local RAM / VRAM | |
| | (DeepSeek / Qwen) | |
| +---------------------+ |
+--------------------------------------------------------+
Step 1: Setting Up the Local Inference Engine (Ollama)
First, we need to install Ollama. It is an open-source tool that packages model weights, configurations, and the CPU/GPU execution engine into a single system daemon.
If you are on macOS or Linux, you can install it via a single terminal command:
curl -fsSL https://ollama.com/install.sh | shFor Windows users, you can download the installer directly from the official Ollama website.
Once installed, verify that the daemon is running by querying the version:
ollama --versionChoosing and Pulling the Right Code Model
Not all LLMs are created equal. For general chat, Llama 3 is fantastic. But for inline code completion (autocompletion as you type), we need a model that supports Fill-In-The-Middle (FIM). FIM allows the model to look at the code before and after your cursor to predict what goes in between.
Two of the best performing, lightweight models for this right now are:
- Qwen2.5-Coder (7B or 1.5B): State-of-the-art coding capabilities for its size, highly efficient.
- DeepSeek-Coder (6.7B): An absolute workhorse that punch above its weight, often rivaling GPT-3.5 on coding tasks.
Let's pull down the 1.5B version of Qwen2.5-Coder for fast autocompletions, and the 7B version for complex refactoring/chat tasks. Run these commands in your terminal:
# For ultra-fast inline autocomplete (low memory footprint)
ollama pull qwen2.5-coder:1.5b
# For chat, code review, and complex explanations
ollama pull qwen2.5-coder:7bStep 2: Configuring VS Code for Local AI
To connect VS Code to our local Ollama server, we will use Continue—an incredible, open-source AI assistant extension that serves as a drop-in replacement for GitHub Copilot.
1. Install the Extension
Open VS Code, navigate to the Extensions Marketplace (Ctrl+Shift+X or Cmd+Shift+X), search for Continue, and install it.
2. Configure Continue to Use Ollama
Once installed, Continue creates a configuration file located at ~/.continue/config.json. We want to configure it to route chat requests to our 7B model and inline tab-completions to our ultra-fast 1.5B model.
Open your ~/.continue/config.json file and update it to look like this:
{
"models": [
{
"title": "Qwen 2.5 Coder 7B (Local)",
"provider": "ollama",
"model": "qwen2.5-coder:7b"
}
],
"tabAutocompleteModel": {
"title": "Qwen 2.5 Coder 1.5B (Local)",
"provider": "ollama",
"model": "qwen2.5-coder:1.5b"
"apiBase": "http://localhost:11434"
},
"customCommands": [
{
"name": "test",
"prompt": "{{{ input }}}\n\nWrite unit tests for the code above, ensuring edge cases are covered.",
"description": "Write unit tests for highlighted code"
}
],
"contextProviders": [
{
"name": "code",
"params": {}
},
{
"name": "docs",
"params": {}
}
]
}Save the file. Restart VS Code, and you will notice a side panel for Continue. Start typing code in any file, and you will see greyed-out inline completions served locally by your machine. Hit Tab to accept them, just like Copilot!
Step 3: Configuring Neovim for Local AI (The Lua Way)
For the Vim/Neovim purists out there, we can build an incredibly sleek, lightweight integration without bloated Node.js runtimes. We will use ollama.nvim or gen.nvim, but my absolute favorite for full Copilot-parity is codecompanion.nvim.
Here is how to set up codecompanion.nvim using lazy.nvim as your package manager. Add the following Lua configuration to your Neovim config (typically in ~/.config/nvim/lua/plugins/ai.lua):
return {
{
"olimorris/codecompanion.nvim",
dependencies = {
"nvim-lua/plenary.nvim",
"nvim-treesitter/nvim-treesitter",
{ "hrsh7th/nvim-cmp", optional = true }, -- Optional: For autocompletion UI
},
config = function()
require("codecompanion").setup({
adapters = {
ollama_chat = function()
return require("codecompanion.adapters").extend("ollama", {
name = "ollama_chat",
schema = {
model = {
default = "qwen2.5-coder:7b",
},
},
})
end,
ollama_inline = function()
return require("codecompanion.adapters").extend("ollama", {
name = "ollama_inline",
schema = {
model = {
default = "qwen2.5-coder:1.5b",
},
},
})
end,
},
strategies = {
chat = {
adapter = "ollama_chat",
},
inline = {
adapter = "ollama_inline",
},
agent = {
adapter = "ollama_chat",
},
},
})
-- Keymaps for easy interaction
vim.api.nvim_set_keymap("n", "<leader>cc", "<cmd>CodeCompanionToggle<cr>", { noremap = true, silent = true })
vim.api.nvim_set_keymap("v", "<leader>cc", "<cmd>CodeCompanionToggle<cr>", { noremap = true, silent = true })
vim.api.nvim_set_keymap("v", "<leader>ca", "<cmd>CodeCompanionActions<cr>", { noremap = true, silent = true })
end,
}
}With this setup, pressing <leader>cc in Normal or Visual mode will open a split window where you can chat with your local model, ask it to refactor code, generate docs, or debug errors—all without your data ever hitting an external server.
Performance Tuning: Getting the Best Out of Your Hardware
Running local models is computationally heavy. If you are experiencing sluggishness or high system heat, here are a few diagnostic steps and tweaks to keep your setup running efficiently:
1. Match Model Size to System RAM/VRAM
As a rule of thumb, you need enough free RAM/VRAM to hold the model weights plus context space. Use this guide to choose your model size:
- 8GB RAM (Standard M1/Intel/AMD): Stick to models under 3 Billion parameters (e.g.,
qwen2.5-coder:1.5b). - 16GB Unified Memory / VRAM: Comfortable running 7B-8B parameter models (e.g.,
qwen2.5-coder:7b,llama3:8b). - 32GB+ Unified Memory / VRAM: Can comfortably run 14B to 32B models (e.g.,
qwen2.5-coder:14bordeepseek-coder:33b).
2. Keep Ollama in System Memory
By default, Ollama unloads models from memory after 5 minutes of inactivity. If you hate waiting for the model to re-load on your first keystroke after a break, you can set an environment variable to keep it loaded indefinitely.
On Linux/macOS, export this variable in your shell profile (.zshrc or .bashrc):
export OLLAMA_KEEP_ALIVE="24h"Wrapping Up: The Local-First Era is Here
Setting up your own local AI copilot isn't just about saving $10 or $20 a month on a subscription. It’s about taking back ownership of your development environment, protecting your intellectual property, and configuring a workspace that works seamlessly wherever you are in the world.
The open-source community is moving incredibly fast. Models like Qwen2.5-Coder and DeepSeek-Coder have narrowed the gap with proprietary models to the point where, for daily coding, refactoring, and boilerplate generation, you won't even notice the difference in quality—but you will notice the speed and privacy.
What about you? Have you made the switch to local LLMs for your coding workflows? Are you team VS Code with Continue, or team Neovim with Lua? Let me know in the comments below, or share your custom configurations!
Until next time, happy hacking!