A language model that runs on your own Mac has no per-token bill, works offline, and never sends your code or documents to anyone else's server. The cost is size. You will not run the largest hosted models, so the job is to pick the biggest model your memory allows and then shape it to your work.
The examples assume an Apple Silicon Mac with 16 GB of memory. Everything scales up or down from there, and the table in the next section shows how.
What Your Mac Can Hold
Apple Silicon uses unified memory: the CPU and GPU share one pool. That is why a laptop can run models that would need a large discrete GPU on a PC, and also why the model competes with your browser and editor for the same RAM.
A model's weights take roughly parameters × bits per weight ÷ 8 bytes. Most local models are quantized to about 4 bits per weight. Once the scaling data is counted, 4-bit formats average a little over 4 bits, so budget about 0.6 GB per billion parameters:
- A 3B model needs about 2 GB and fits comfortably in 8 GB.
- An 8B model needs about 5 GB and fits comfortably in 16 GB.
- A 14B model needs about 8.5 GB and fits comfortably in 24 GB or 32 GB.
- A 32B model needs about 19 GB. It is tight in 32 GB and comfortable in 48 GB.
- A 70B model needs about 42 GB, so plan on 64 GB or more.
These are my own arithmetic, not benchmarks, and the memory sizes leave room for macOS and your other apps. macOS also limits how much unified memory the GPU can use, so you never get the full amount. On a 16 GB Mac, an 8B model at 4-bit is the comfortable choice, and a 14B model is possible with everything else closed.
Context length costs memory too. The model keeps a cache of every token in the conversation, and it grows with the context window. A model that fits at 4,000 tokens of context can run out of room at 32,000.
To see what you have:
sysctl -n hw.memsize | awk '{print $1 / 1024 / 1024 / 1024 " GB"}'
Running a Model with Ollama
Ollama is the quickest way to start. It downloads models, runs them on the GPU through Metal, and exposes a local HTTP API.
brew install ollama
brew services start ollama
ollama run llama3.2:3b
ollama run pulls the model on first use and opens a chat in the terminal. The commands I reach for after that:
ollama list # models on disk
ollama ps # models loaded in memory, and whether they run on the GPU
ollama pull qwen2.5-coder:7b
ollama rm llama3.2:3b
Model names change as new versions ship, so check the Ollama library for current tags and sizes before you pull.
The server listens on localhost:11434 and also speaks the OpenAI chat format, which means most client libraries work if you change the base URL:
const res = await fetch("http://localhost:11434/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "llama3.2:3b",
messages: [{ role: "user", content: "Explain a mutex in one sentence." }],
}),
});
const data = await res.json();
console.log(data.choices[0].message.content);
Ollama unloads a model after a few minutes idle, so the first request after a break is slower while the weights load again. That delay is loading, not generation. Set OLLAMA_KEEP_ALIVE if you want it to stay resident.
Other Ways to Run a Model
Ollama is not the only option, and the others are worth knowing for what they do differently.
- llama.cpp (
brew install llama.cpp) is the engine many tools are built on. It gives you direct control over quantization, context size and GPU layers, andllama-serverprovides the same kind of HTTP API. - MLX is Apple's array framework for its own chips, and
mlx-lmruns and fine-tunes models with it. It is the path used for fine-tuning below. - LM Studio is a desktop app with a model browser and a chat window. It is a good fit if you would rather not use a terminal.
All three read widely shared model formats, so trying one doesn't lock you out of the others.
Making the Model Your Own
Once a general model runs, the question is what to change. There are three levels, and each costs more than the last. Start at the first and move down only when it isn't enough.
Level 1: A Modelfile
A Modelfile wraps an existing model with a system prompt and default parameters. Nothing is trained. This one turns a base model into a code reviewer:
FROM qwen2.5-coder:7b
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM """
You review pull requests for Go and Laravel code.
Answer with a short list of concrete problems, most serious first.
Quote the line you mean. If the change looks fine, say so in one sentence.
"""
Build and use it:
ollama create reviewer -f Modelfile
ollama run reviewer "Review this: $(git diff HEAD~1)"
ollama create reuses the base model's weights on disk, so the new model costs almost no extra space. Setting num_ctx yourself matters because the default context window is small and long inputs get cut off without warning. A low temperature keeps review output steadier from run to run.
Level 2: Your Documents
When people say they want a model trained on their notes, they usually want retrieval, not training. You split your documents into chunks, turn each chunk into a vector with an embedding model (Ollama can serve one, for example nomic-embed-text), and store the vectors. When you ask a question, you look up the closest chunks and paste them into the prompt.
The model never learns the documents. It reads the relevant ones at question time, which is why you can add a file and have it used in the next answer, and why the answer can point back to the source. For facts that change or that must be traceable, this beats fine-tuning.
Level 3: A LoRA Fine-Tune
Fine-tuning changes the model itself. LoRA does it cheaply: it freezes the original weights and trains a small set of extra ones, so a laptop can do it on a small model.
LoRA is good at format, tone and narrow tasks, such as always answering in your review style or producing a fixed JSON shape. It is a poor way to add facts, because the model picks up the style of your examples much more reliably than their content.
Set up MLX in a virtual environment:
python3 -m venv .venv
source .venv/bin/activate
pip install mlx-lm
Training data goes in a folder with train.jsonl and valid.jsonl, one example per line:
{"messages": [{"role": "user", "content": "Review: if err != nil { return nil }"}, {"role": "assistant", "content": "1. The error is swallowed. Return it or wrap it with context."}]}
Start with a few hundred examples you have checked by hand. A handful mostly teaches the model to repeat them, and examples that disagree with each other teach it to be inconsistent.
Train a 4-bit base model, which keeps memory low:
mlx_lm.lora \
--model mlx-community/Llama-3.2-3B-Instruct-4bit \
--train \
--data ./data \
--iters 600 \
--batch-size 1
Training uses more memory than inference. If it runs out on a 16 GB Mac, lower --batch-size, reduce --num-layers, or add --grad-checkpoint. The adapter is saved to adapters/ by default. Try it before you commit to it:
mlx_lm.generate \
--model mlx-community/Llama-3.2-3B-Instruct-4bit \
--adapter-path adapters \
--prompt "Review: defer file.Close() inside a loop"
If the answers improve, fuse the adapter into a standalone model and serve it:
mlx_lm.fuse \
--model mlx-community/Llama-3.2-3B-Instruct-4bit \
--adapter-path adapters \
--save-path fused
mlx_lm.server --model ./fused --port 8080
mlx_lm.server offers an OpenAI-style API like Ollama's. Ollama can also import a fused model from a local folder with FROM ./fused in a Modelfile, but only for architectures it supports, so check that before you plan around it. Model repository names on Hugging Face change, so look through the mlx-community page for a current 4-bit build.
Mistakes That Cost Time
Choosing by Parameter Count Alone
A 14B model squeezed into too little memory loses to an 8B model that fits with room to spare. Quantization matters as much as size, and very aggressive settings, around 2 bits, hurt small models noticeably. Leave a few gigabytes free.
Running Past Your Memory
When the model and its cache don't fit, macOS starts swapping and generation slows to a crawl. Run ollama ps while a model is loaded. If the processor column shows a CPU and GPU split, part of the model didn't fit on the GPU, and a smaller model or shorter context will be faster than the larger one.
Exposing the Server
Ollama binds to localhost by default and has no authentication. Setting OLLAMA_HOST=0.0.0.0 to reach it from another device makes it reachable by everyone on that network. Put it behind a reverse proxy with authentication, or a private network like Tailscale, before you do.
Judging a Model by Its First Answer
The first request is slow because the weights are loading, and one answer says little about quality. Run the same ten prompts against each candidate and compare them side by side.
Fine-Tuning for Facts
If the model needs to know something specific and current, use retrieval. Fine-tune when the problem is how it answers, not what it knows.
Heat
A laptop that runs a model for a long stretch gets hot and slows itself down. A MacBook Air has no fan, so expect speed to drop in long sessions.
Summary
- Unified memory is the limit. Budget about 0.6 GB per billion parameters at 4-bit, plus room for context and macOS.
- Ollama is the fastest start, and its OpenAI-compatible API works with existing clients.
- Customize in order: a Modelfile first, retrieval for your documents, a LoRA fine-tune last.
- Fine-tune to change style and format. Use retrieval to supply facts.
- Check
ollama pswhen things are slow, and keep the server on localhost.
The Ollama Modelfile reference and the mlx-lm LoRA guide list every option used here and many more.