If you’ve been using AI tools for a while, you’ve probably run into this at some point: you find an open source model that looks amazing, you want to try it on your own machine, and ten minutes later you’re stuck in the same mess as always. You don’t know if your GPU can handle it, you don’t know how much VRAM you really need, and you end up downloading something so heavy your computer starts begging for mercy.
Running models locally sounds great, and sometimes it really is worth it. You get more privacy, you control the cost, you don’t depend on an external API, and you can build surprisingly capable workflows on your own hardware. But there is also a lot of nonsense around this topic. Not everyone needs local AI, and not every machine is built to run certain models without suffering.
In this post we’re going to do it properly. First we’ll check whether your machine can handle it with CanIRun.ai, then install Ollama, pull a model, test it locally, and from there connect it to tools like Open Code or Cloud Code if you want that model inside your daily workflow.
Why you might want to run AI locally
The first reason is usually the obvious one: privacy. If you’re working with code, notes, internal documents, or you simply don’t love sending everything to a third party, running the model on your own machine gives you a very different level of control.
The second reason is cost. The moment you start using APIs seriously, especially for coding or heavy iteration, the bill stops being cute. Locally you pay in hardware and electricity, sure, but in exchange you can experiment without staring at a usage meter every five minutes.
And the third reason, which to me is one of the most interesting ones, is control. You can choose the exact model you want, switch quantizations, try one fine-tuned for code, another for reasoning, another for RAG, and build a workflow that actually fits the way you work. You’re not depending on a company changing the model, adding weird limits, or killing a feature overnight.
That said, one important point: local does not always mean better. If your laptop is modest, or you’re trying to run a 30B model as if physics were optional, reality is going to hit hard. That’s why you should start with the thing most people ignore.
Before downloading anything, check whether your machine can handle it
The fastest way to stop wasting time is to go to CanIRun.ai. That site detects your machine and gives you genuinely useful estimates of which models should run well, which ones are a tight fit, and which ones are simply a bad idea.
The good part is that it doesn’t stop at the model name. It shows you VRAM, context size, and different quantizations, which is exactly what you need to look at if you don’t want to go in blind.
If CanIRun.ai marks a model as Runs great or Runs well, great. If it says Tight fit or Barely runs, that doesn’t mean impossible, but it is warning you that you’re probably going to sacrifice speed, context, or stability. And if it tells you the model is too heavy, don’t try to get clever. There is nothing wrong with starting with a smaller model. A lot of people want to begin with something huge because it sounds cooler on social media. That’s not engineering. That’s ego.
VRAM is what really decides this
This is one of the key ideas. When we talk about running models locally, what matters is not just whether you have a GPU, but how much VRAM that GPU has.
VRAM is your graphics memory. Ideally, that is where you want the model weights and a big part of inference work to live. If the model fits comfortably there, the experience changes completely. If it doesn’t, the compromises begin: some of it spills into RAM, some gets pushed to CPU, latency jumps, and the thing you hoped to use every day becomes a pain.
That is why it is not enough to think “this model is 7B” or “that one is 14B.” Quantization matters too. The same model can take far less space in Q4_K_M than in Q8_0 or F16, but of course that trade-off is not free: you’re balancing memory, quality, and performance.
The easiest way to think about it is this:
- If VRAM is tight, you want smaller models or more aggressive quantizations.
- If you have plenty of VRAM, you can afford larger models or more comfortable quantizations.
- If a model “almost fits,” the real experience is often worse than people expect.
And watch out for one more thing: VRAM does not only affect whether a model starts. It also affects how much context you can use and the real speed once you begin feeding it long prompts, tools, or coding workflows. A model that answers one short question decently can fall apart the moment you ask it to do real work.
Note: CanIRun.ai gives you a very useful estimate, but it is still an estimate. Treat it as a first filter, not as divine truth carved in stone.
Install Ollama without overcomplicating it
If your goal is to run models locally without turning it into a science project, Ollama is one of the easiest options right now. Install it, pull a model, and move on. No weird runtime gymnastics, no unnecessary drama.
On Linux or macOS you can install it like this:
curl -fsSL https://ollama.com/install.sh | shIf you’re on Windows, the sensible move is to download the installer from the official site or use WSL if your workflow already lives there.
Once it’s installed, verify it:
ollama --versionAnd if you want to check that the local service is up and exposing its OpenAI-compatible endpoint, run this:
curl http://localhost:11434/v1/modelsThat detail matters, because it is exactly what lets you connect Ollama to other tools later.
Pull a model with ollama pull
This is where judgment matters. Do not start with the biggest model you saw in a benchmark thread. Start with something your machine can actually run.
For coding or general testing, a pretty sensible place to start is this:
ollama pull qwen2.5-coder:7bIf you want something more general-purpose, you can also look at families like qwen3:8b, llama3.1:8b, or a smaller Gemma model, but the core idea is the same: pick based on your hardware, not on hype.
When the download finishes, list your local models:
ollama listThat gives you the exact model name you’ll need later in CLI tools or external clients.
Test it locally before wiring it into other tools
This matters more than people think. Before you plug the model into Open Code, Cloud Code, or any agent setup, test it directly. That way you can see whether it responds well, whether it’s too slow, or whether your machine starts struggling immediately.
The simplest test is this:
ollama run qwen2.5-coder:7bOnce you’re inside, ask it something basic but useful, for example:
Write a TypeScript debounce function and explain why it works.If it already feels too slow here, don’t expect miracles once you shove it inside a tool-driven workflow with long context and real code. Local AI looks great in a demo, but what matters is whether it is actually useful in your day-to-day work.
How to use it later with Open Code and Cloud Code
This is the fun part. Ollama exposes OpenAI-compatible API support at http://localhost:11434/v1, and that opens the door to any client that supports OpenAI-compatible endpoints.
In Open Code, this fits nicely because it lets you configure custom providers and local models. A configuration example would look like this:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"ollama": {
"npm": "@ai-sdk/openai-compatible",
"name": "Ollama (local)",
"options": {
"baseURL": "http://localhost:11434/v1"
},
"models": {
"qwen2.5-coder:7b": {
"name": "Qwen 2.5 Coder 7B (local)"
}
}
}
}
}With that in place, Open Code can talk to your local model just like any other provider, except the whole thing stays on your machine.
In Cloud Code, or any similar client, the idea is the same if it lets you configure an OpenAI-compatible provider or a custom baseURL. If it does, point it to:
http://localhost:11434/v1Then use the model name you already loaded into Ollama.
If the tool doesn’t let you change the endpoint and only works with the provider’s locked-down APIs, then you are not going to connect Ollama directly, full stop. Better to say that clearly. Sometimes the problem is not your machine. The client just was not designed for that workflow.
Keep your expectations realistic: local is useful, not magical
This is the part the internet loves to gloss over. If you have a decent GPU and you choose the right model, you can absolutely build a setup that is genuinely useful. But don’t expect a small local model to perform like the best cloud model of the moment on hard reasoning, architecture work, or complex agent tasks.
Where local usually makes a lot of sense is when you’re iterating quickly, you care about privacy, you want to automate very specific things, you repeat the same tasks over and over, you’re building prototypes, or you simply want tighter control over cost.
Where you need to calm down a bit is when someone tries to use an undersized local model as a total replacement for everything. If your hardware is limited, you can still do useful work, but with trade-offs. Less speed, less comfortable context, and a lot more need to choose the right use case.
And that’s fine. In fact, using local AI intelligently usually comes down to exactly that: knowing which tasks deserve to stay local and which ones don’t.
What if I want to use fine-tuned models
You can, and in many cases it makes a lot of sense.
If you find a fine-tuned model for code, SQL, support, RAG, or whatever you do all the time, you can often get behavior that feels more focused than a generic base model. But don’t confuse “fine-tuned” with “miraculous.” If the base model is weak or the hardware is choking, tuning will not save you from reality.
That is the beauty of Ollama and the open ecosystem: you can try several options and keep the one that gives you the best balance between quality, memory use, and speed for your actual work.
The takeaway I would keep
If you want to try local AI, don’t start by downloading random things and hoping for the best. Start with CanIRun.ai, understand how much VRAM you really have, choose a model that fits your machine, and get something simple running first with Ollama.
Once that works well, then yes, connect it to Open Code, Cloud Code, or whichever client makes sense for your workflow. But in that order. First real compatibility, then model choice, then integration. Not the other way around.
Because if you do it backwards, you’ll probably end up saying local AI is terrible, when what was actually terrible was the approach.
If you want, in another post I can take this one step further and build concrete setups based on the type of machine you have: a regular laptop, a desktop with a dedicated GPU, or a mini home server.
What do you think?
Leave your opinion, question or suggestion. Comments are synced with GitHub Discussions .