LLMs and RAG for product managers
The two ideas that shape most AI products, explained without the maths: what a language model is, why RAG matters, and the tradeoffs you will actually make.
You do not need to build models to be an AI product manager, but you do need to understand the handful of ideas that shape what your product can do. LLMs and RAG are the two that come up most. This is the working mental model, without the maths.
What a large language model actually is
A large language model, or LLM, is a system trained on huge amounts of text to predict the next piece of text given what came before. That simple mechanism, at scale, produces something that can answer questions, write, summarise and reason in useful ways. The key product consequences follow from how it works: it is probabilistic, it can be confidently wrong, and its knowledge is frozen at training time unless you give it fresh information.
Why RAG exists
RAG stands for retrieval augmented generation. Because a model does not know your private data or anything after its training cutoff, RAG fetches relevant documents at the moment of the question and hands them to the model as context, so it can answer using current, specific information. For most business AI products, RAG is the difference between a clever demo and something that actually knows about your company.
Fine-tuning, prompting and RAG
| Approach | What it does | Use when |
|---|---|---|
| Prompting | Instruct the model well at request time | Almost always the first thing to try |
| RAG | Give the model your data as context | The answer depends on private or fresh information |
| Fine-tuning | Adjust the model on your examples | You need a specific style or behaviour prompting cannot reach |
The tradeoffs you will actually decide
Every AI feature lives inside a triangle of accuracy, latency and cost. A bigger model is often more accurate but slower and more expensive. Your job is to pick the point on that triangle that fits the user context, then design around it. A support copilot can trade a little speed for accuracy; a live typeahead cannot.
- Accuracy: how often the output is good enough for the user.
- Latency: how long the user waits, which shapes whether the feature feels usable.
- Cost: real money per request, which shapes pricing and which model you can afford.
Frequently asked questions
What is an LLM in simple terms?
A large language model is a system trained on huge amounts of text to predict the next piece of text. At scale that lets it answer, write, summarise and reason. For product work the key facts are that it is probabilistic, can be confidently wrong, and only knows what it saw in training unless you give it fresh context.
What is RAG and why does it matter?
RAG, retrieval augmented generation, fetches relevant documents at the moment of a question and gives them to the model as context so it can answer using current or private information. It is what lets an AI product answer accurately about your specific company data rather than only general knowledge.
Should I fine-tune or use RAG?
Start with good prompting, add RAG when the answer depends on private or fresh data, and fine-tune only when you need a specific style or behaviour that prompting cannot reach. Most business AI products get most of their value from RAG plus strong prompting, not fine-tuning.
Do product managers need to understand model tradeoffs?
Yes. Every AI feature sits in a triangle of accuracy, latency and cost, and choosing the right point is a product decision, not just an engineering one. Knowing the tradeoffs lets you pick the right model and design the experience around its limits.
Do I need to code to understand LLMs and RAG?
No. You need the working mental model: what the model does, why RAG exists, and the accuracy, latency and cost tradeoffs. That fluency is enough to make product decisions and talk credibly with engineers, without writing model code yourself.