Photo by Editor
# LLM Problem(s) Addressed
“Start your master language model (LLM)” “just start your business” for 2026. It sounds like a dream: no API costs, no data leaving your servers, full control over the model. Then you actually do it, and reality starts to show up uninvited. The GPU is running out of central memory. The model looks worse than the managed version. Delay is shameful. Somehow, you've spent three weekends on something that still can't answer basic questions.
This article is about what really happens when you take self-employed LLMs seriously: not benchmarks, not hype, but the real conflict of many teaching practices completely bypassed.
# Hardware Reality Check
Most tutorials just assume you have a physical GPU lying around. The fact is that using the 7B parameter model comfortably requires at least 16GB of VRAM, and if you push it towards the 13B or 70B area, you are looking at a multi-GPU setup or a significant trade-off of speed quality through scaling. Cloud GPUs help, but you're back to paying per token on a rolling basis.
The gap between “fast” and “efficient” is wider than most people expect. And if you're targeting anything near-production, “running” is the worst place to stop. Infrastructure decisions made at the beginning of a hosting project have a way of compounding, and changing them over time is painful.
# Quantization: Saving Grace or Compromise?
Quantization is a common practice for hardware problems, and it's worth understanding what you're actually trading. When you downgrade the model from FP16 to INT4, you greatly compress the weight representation. The model becomes faster and thinner, but the accuracy of its internal calculations decreases in ways not always seen before.
In a general purpose discussion or summary, a lower estimate is usually correct. Where it starts to creep in is in thinking tasks, producing structured output, and anything that requires careful instruction. A model that handles JSON output reliably in FP16 may start generating broken schemas in Q4.
There is no universal answer, but the methodology is very powerful: test your specific use case at all levels of measurement before committing. Patterns usually emerge quickly once you've used enough information in both versions.
# Context Windows and Memory: The Invisible Ceiling
One thing that catches people off guard is how quickly context windows fill the workflow, especially if you have to scale while using. Ollama. A 4K content window sounds good until you build a recovery generation pipeline (RAG) and suddenly you're injecting system information, recovery components, chat history, and an actual user query all at once. That window is disappearing faster than expected.
Long context models exist, but using a 32K context window at full attention is computationally expensive. Memory usage averages about four times the length of a context under normal attention, meaning that doubling your context window can quadruple your memory requirements.
Effective solutions include dynamic filtering, trimming chat history, and being more selective about what goes into context. It's not as good as having unlimited memory, but it forces a kind of fast instruction that tends to improve output quality anyway.
# Latency Is A Feedback Loop Killer
Hosted models are often slower than their API counterparts, and this is more important than people first thought. If an idea takes 10 to 15 seconds to get a decent response, the improvement loop is significantly shortened. Test prompts, iterating on output formats, debugging chains — everything gets bogged down in waiting.
Streaming responses improve the user-facing experience, but do not reduce the overall time to completion. For background or batch jobs, the delay is less important. For anything interactive, it becomes a real usability problem. Reliable performance is an investment: better hardware, improved deployment frameworks such as vLLM or Ollama with proper configuration, or combining applications where the workflow allows. One of these is the cost of owning the stack.
# Fast Behavior Moves Between Models
Here's something that affects almost everyone who switches from hosting to self-hosting: instant templates are very important, and they are model-specific. A system command that works well with a hosted boundary model may produce inconsistent output from Mistral or LLaMA fine-tune. Models are not broken; they are trained in different formats and respond accordingly.
Each model family has its own expected teaching structure. The LLaMA models were trained with Alpaca format expects one pattern, dialog-triggered models expect another, and if you use the wrong template, you get a confused attempt by the model to respond to incorrect input instead of an actual failure of the skill. Most frameworks handle this automatically, but it's worth checking manually. If the output sounds oddly off or inconsistent, a quick template is the first thing to check.
# Fine Tuning Sounds Easy Until It Isn't
Sometimes, most of the players themselves think about fine tuning. The basic model handles the general case fine, but there is a particular background, tone, or structure of work that can really benefit from a model trained on your data. It makes sense in theory. You wouldn't use the same model for financial calculations as you would for writing three.js animations, right? Of course not.
Therefore, I believe that the future will not be that Google will suddenly release a model like Opus 4.6 that can work on NVIDIA's 40 series card. Instead, we will see models designed for specific niches, functions, and applications – resulting in fewer parameters and a better allocation of resources.
To do well, to adjust well or with LoRA or QLoRA it requires clean and well-structured training data, rational computation, careful hyperparameter selection, and a reliable test setup. Most initial attempts generate a model that is not confident about your domain in ways that the base model was not.
A lesson many people learn the hard way is that data quality is more important than data quantity. A few hundred carefully selected examples will often outnumber noisy thousands. It's a boring job, and there's no shortcut to it.
# Final thoughts
Self-taught LLM is possible at the same time and is more difficult than advertised. The use of tools has been really good: Ollama, vLLM, and the wider ecosystem with an open model have lowered the barrier in a meaningful way.
But the hardware cost, quantization trade-off, speed bump, and fine-tuning curve are all real. Go in expecting a frictionless hosted API replacement and you'll be disappointed. Go in expecting to own a system that rewards patience and repetition, and the picture looks much better. Difficult lessons are not distractions in this process. They are a process.
Here is Davies is a software developer and technical writer. Before devoting his career full-time to technical writing, he managed—among other interesting things—to work as a lead programmer at Inc. 5,000 branding whose clients include Samsung, Time Warner, Netflix, and Sony.
