Writing ·
How to Choose an LLM for a Product Feature
By Sovaid Shah, Software Architect · AI · LLM · OpenAI · Claude · Gemini · Architecture
TL;DR: Do not pick an LLM from leaderboards or launch posts. Write down what the feature must do, build an evaluation set of 30 to 50 real examples, and test two or three models on it. Compare quality on your examples, latency, cost per request, context size and the provider data policy. Start with the smallest model that passes, and keep the model name in configuration so you can switch later.
“Which model should we use?” is usually the first question a team asks about an AI feature. It should be about the fourth. Models change every few months, and the best model on a public benchmark is often not the best one for your specific task, budget and latency.
This is the process I’d use to choose, whichever providers are on the shortlist.
Step 1: define the job precisely
Write one paragraph that says:
- Input: what goes in (a support email, a 40-page contract, a product description)
- Output: what comes out, and in what format (three bullets, a JSON object, a yes/no with a reason)
- Quality bar: what “good enough” means, and what a bad answer costs (a slightly awkward summary vs. wrong legal advice)
- Speed: whether a user is waiting live, or it runs in the background
- Volume: roughly how many requests a day
This paragraph rules out options fast. A live autocomplete feature and an overnight document classifier need very different models.
Step 2: build an evaluation set
Collect 30 to 50 real examples of input with known good output. Include the hard cases: long inputs, messy formatting, edge cases and examples where the right answer is “I can’t tell”.
This is the most valuable hour you’ll spend on the feature. Every model comparison, prompt change and future model upgrade gets tested against it.
Step 3: compare on five things
Test two or three models from different providers (for example OpenAI, Anthropic’s Claude and Google’s Gemini, or an open-weights model if you need to self-host). Compare:
| Factor | What to check |
|---|---|
| Quality | Correct answers on your eval set, in the format you need |
| Latency | Time to first token and total time, for realistic input sizes |
| Cost | Price per request at your typical input and output length |
| Context | Can it take your largest realistic input, with room to spare? |
| Data policy | Data retention, training use, and where data is processed |
Quality often turns out closer than expected between top models on a specific task, and then cost and latency decide it.
Step 4: start smaller than you think
Providers offer large, capable models and smaller, faster, cheaper ones. For narrow, well-defined tasks such as classification, extraction and short summaries, smaller models often pass the eval set at a fraction of the cost and with lower latency.
Start with the smallest model that passes. Move up only for the cases that fail. Some teams route requests: a small model handles most, and hard or low-confidence cases go to a larger one.
Step 5: don’t lock yourself in
- Keep the model name in configuration, not hard-coded.
- Put provider calls behind one internal function or module, so switching providers touches one place.
- Re-run your eval set when a provider releases a new model. Sometimes it’s an easy upgrade. Sometimes it quietly breaks your output format.
Open models vs hosted APIs
Hosted APIs are the fastest way to start: no GPUs, no scaling work. Open-weights models make sense when data can’t leave your infrastructure, when volume is high enough that self-hosting is cheaper, or when you need to fine-tune heavily. They come with real operating costs: servers, scaling, monitoring and updates. Run the numbers before assuming self-hosting is cheaper.
A note on data and privacy
Before sending customer data to any provider, read their current data usage terms. Check whether API data is used for training, how long it’s kept, and where it’s processed. For healthcare, finance or EU personal data, involve whoever owns compliance early. It’s much harder to change providers after launch.
FAQ
Which is better, OpenAI, Claude or Gemini?
It depends on the task. Test each on 30 to 50 real examples from your product and compare quality, latency and cost. Public benchmarks rarely match a specific product feature.
Should I use the biggest model available?
Usually not to start. Smaller models are often good enough for narrow tasks, and they’re faster and cheaper. Use larger models only where your evaluation shows they’re needed.
How do I avoid vendor lock-in with LLMs?
Keep model names in configuration, put provider calls behind one internal module, and keep an evaluation set so you can test alternatives quickly.
Related: how to integrate an LLM API in Next.js and LLM features with background jobs. Planning an AI feature? See software architecture.