There is a question I get from business owners now that I did not get a year ago: is there a way to use AI without sending our information to somebody else’s computer?
Yes. You can run a capable AI model on hardware sitting in your own office. I do it. Whether you should is a different question, and it depends on things most articles about this skip past.
This is the honest version.
What “local” actually means
A local model is a language model that runs on your own machine instead of in a provider’s data center. You install software such as LM Studio or Ollama, download a model file, and it answers from your hardware. No account, no per-token bill, no network request leaving the building.
I run one on a desktop with a strong graphics card. It is not a frontier model and it does not pretend to be, but it drafts, summarizes, rewrites, answers questions about documents I give it, and helps with scripting. For a meaningful share of everyday work, the gap between it and a paid cloud model is smaller than people expect.
The three reasons a small business would want this
1. The data never leaves
This is the real one. In my guide to AI-built tools, the third failure pattern is business data getting pasted into AI services with nobody checking where it is retained. Customer records, invoices, employee files, patient names.
A local model removes the question entirely. There is nothing to check, because nothing was transmitted. If you are subject to a regulation that cares where data lives, that distinction is worth more than any feature comparison.
2. The cost is a purchase, not a meter
Frontier AI is currently cheap because investors are paying for a lot of it. The providers are spending enormous amounts to keep these systems running while the market sorts itself out.
I do not know when that changes or by how much. Nobody does. But planning your operations around a price that is being subsidized during a land grab is a risk worth naming out loud. Hardware you own has a known cost and does not reprice.
3. It works when the internet does not
Smaller point, but a real one for anyone who has watched a business stop because a service was down.
What it actually takes
The honest constraint is graphics memory, not money.
Rough guide, using the compressed model files most people run:
- Around 8 GB of video memory runs small models well. Good for drafting, summarizing, and simple questions.
- 12 to 16 GB opens up mid-sized models. This is where it starts feeling genuinely useful for real work.
- 24 GB and up runs the larger open models, and gets closest to what people expect from a paid service.
Treat those as approximate. Model efficiency keeps improving, and what needed 24 GB last year often fits in less now.
Practically: a decent gaming graphics card is the entry point. Plenty of businesses already have a machine that qualifies, or can get there for the price of about a year of team AI subscriptions. This is one of the few cases where a well-specified gaming PC is a legitimate business purchase.
What local models are genuinely bad at
I am not going to sell you something that does not work.
- The hardest reasoning. For complicated multi-step problems, the big commercial models are still clearly better. This is not close.
- Very long documents. Local models handle less context at once. A 200-page contract is not the job for this.
- Speed on weak hardware. Undersized graphics memory means the model runs on the processor instead, and it becomes slow enough that people stop using it.
- Setup that stays working. Someone has to install it, pick a model, keep it updated, and fix it when it breaks. That someone is either you or a person you pay.
A local model is a competent generalist, not a replacement for a specialist. The businesses that get value from this use it for the eighty percent that is routine and keep a paid account for the hard twenty percent, with a rule about what may go into the paid one.
When not to do this
Skip it if any of these are true:
- Nobody at your company wants to own the maintenance.
- Your AI use is occasional. A subscription is cheaper than hardware you rarely use.
- The work you need is at the top of the difficulty range.
- You have no policy about what data goes where. Fix that first. A local model does not help if staff keep pasting client files into whatever is open in the browser.
That last one matters most. Running your own model is a technical control layered on top of a decision about handling data. Without the decision, the control is decoration.
A short decision checklist
- What specific information are you trying to keep in-house? Name it.
- How often would this actually get used, honestly?
- Do you already own a machine with a capable graphics card?
- Who maintains it, and what happens when that person is on vacation?
- What stays on the paid service, and who decides?
Three or more clear answers means it is worth pricing out. Fewer means fix the policy first.
Where this goes next
The direction that interests me is models that do more than answer. Agent software can let a model run commands and complete tasks rather than just producing text, and it runs on a local model perfectly well. I have written about that at length in When AI Gets Hands, along with why I think rising costs push more organizations toward hosting their own.
That is genuinely powerful and genuinely worth being careful with. Same rule as everything else here: understand it before you point it at anything that matters.
Where to go next
The free Small Business Security Checkup covers whether you have clear rules for AI tools and unapproved cloud services, which is the decision that should come before any of this.
If you want to talk through whether local AI makes sense for your situation, or you want help specifying and building the machine, that is what IT Wes is here for.