The last stop on my AI roadmap is optional on purpose. Every earlier stop used models as they come, through an API, and that is genuinely enough to build most things. This stop is for when it is not: when you want to change how a model behaves, run one without a provider, or simply understand what is happening inside. It is the difference between ordering well and learning to cook.
What is actually inside
A language model is a neural network: a very large arrangement of numbers (weights) that turns input tokens into a prediction of the next one. The specific design that made modern models work is the transformer, whose key trick, attention, lets every token weigh how much every other token in the window matters to it. That is the mechanism behind “it understood the whole paragraph”.
Training is how those weights get their values. Pretraining runs the network over an enormous pile of text, adjusting the weights until its next-token guesses are good. That is the expensive, months-long part that only big labs do. What comes after is where you might get involved.
The question everyone asks: should I fine-tune?
Fine-tuning means continuing to train an existing model on your own examples, so its behaviour shifts. It sounds like the obvious way to make a model “yours”, and in 2026 the honest answer to “should I fine-tune” is usually “not yet”. The field’s sequence is prompt first, then retrieval, then fine-tune, then distill, and most teams never need the last two.
flowchart TD
A{Can a better prompt
and examples fix it?} -->|yes| P[Prompt engineering]
A -->|no| B{Is the problem missing
or changing knowledge?}
B -->|yes| R[RAG: retrieve your data]
B -->|no, it is stable behaviour,
format, or tone| F[Fine-tune a thin adapter]
F --> D[Distill to a small model if volume is high]
The rule that resolves it: RAG for knowledge that changes, fine-tuning for behaviour that is stable. Fine-tune when you need a consistent voice, a non-standard output format prompting cannot hold, or a latency budget with no room for a retrieval step. And the modern way is not retraining the whole model. It is a thin adapter (LoRA, QLoRA) trained on top of a strong base, usually kept alongside retrieval rather than replacing it.
TIP
The strongest 2026 case for fine-tuning is distillation: use a large frontier model to generate excellent examples for a task, then train a small open model on them. You get most of the quality at a fraction of the cost per call, for a task the small model already half-handled.
Running a model on your own machine
You do not need a provider to run a model. Open-weight models can be downloaded and run locally, which buys privacy (nothing leaves your machine), no per-token cost, and offline use.
The tool that made this easy is Ollama: one command pulls a model and starts a local server with the same API shape the hosted providers use, so your existing code often works unchanged. It wraps llama.cpp, the lower-level engine, which you reach for when you want fine control.
The reason this fits on a laptop at all is quantization: storing the weights with fewer bits. A 4-bit version cuts memory by around three quarters with little visible quality loss, and a quantized model with 7 billion weights (a “7B” model) runs at a usable 10 to 20 tokens a second on a modern CPU.
ollama pull llama3 # download an open model, quantized for your machine
ollama run llama3 # chat with it locally, nothing sent anywhere
NOTE
Local models are smaller and weaker than the frontier hosted ones. They shine for privacy, cost, and experimentation, and for well-defined tasks a fine-tuned small model handles. For the hardest reasoning, the big hosted models still win.
How models learn to be helpful
One last idea, because you will hear the acronyms. A pretrained model is good at continuing text, not at being a helpful assistant. That behaviour is added afterwards by alignment training. RLHF (Reinforcement Learning from Human Feedback) does it in stages: fine-tune on good examples, train a separate model to score answers the way humans would, then optimise the main model against that score. A newer, simpler method, DPO, skips the separate scoring model and learns directly from pairs of preferred and rejected answers. You are unlikely to run either yourself, but they explain why a model answers like an assistant rather than autocompleting your question.
Going deeper: the maths, if you want it
Three areas underlie all of this. Linear algebra (vectors and matrices are what weights and embeddings literally are), probability (the model outputs a probability over the next token), and enough calculus to understand gradient descent, the method that adjusts weights during training. You can build with models knowing none of it. You need it the moment you want to change one.
The tools you actually reach for
| Job | Popular tools | Note |
|---|---|---|
| Run locally | Ollama, LM Studio, llama.cpp | Free/OSS; Ollama is the easy default |
| Open-weight models | Llama, Mistral, Gemma, Qwen | Free to download from Hugging Face |
| Fine-tuning adapters | Unsloth, Hugging Face PEFT | Free/OSS; LoRA and QLoRA |
| Learning the maths | 3Blue1Brown neural network series, fast.ai | Free |
Order well first, cook only when you must
The lesson from this last stop is a sense of proportion. Understanding the transformer, running a model locally, and knowing what fine-tuning is for all make you better at every earlier stop, even if you never train anything. But the sequence matters: prompt, then retrieve, then, only if behaviour still is not right, fine-tune. Most good AI products are built by people who order well and never open the kitchen. Knowing how the kitchen works is what lets you tell when you finally need to.
Learn how it is made, so you know when to make it.
Sources
- BigDataBoutique: fine-tuning LLMs in 2026, when RAG is not enough for the prompt, RAG, fine-tune, distill sequence and adapters
- daily.dev: running LLMs locally in 2026, Ollama, llama.cpp, self-hosted AI for Ollama, llama.cpp, and quantization
- SitePoint: run local LLMs 2026, complete developer guide for local model performance and 4-bit quantization
- SuperAnnotate: what is direct preference optimization (DPO) for RLHF stages and DPO