TLDR
- Open-weight AI models trail the best closed models by about four months, per Epoch AI.
- Ollama and LM Studio are free tools that run these models on a personal computer.
- Small models need about 8GB of VRAM or Mac unified memory.
- LM Studio can operate fully offline, keeping chats on the user’s machine.
- The largest open models remain far beyond consumer hardware.
Open-weight AI models now trail the best closed models by about four months on average. Epoch AI reported the figure in a May 29 analysis covering the period since January 2026. A growing number of these models run on hardware many people already own.
“Open-weight” means a company publishes the weights, the files that hold what the model learned. Anyone can download and run them. Two free programs, Ollama and LM Studio, handle the setup.
Why Run a Model Locally
A cloud chatbot sends every prompt to a company’s servers. A court ordered OpenAI to preserve ChatGPT logs, including deleted ones, as Decrypt reported in a review of privacy-focused AI tools. A local model keeps the conversation on the user’s computer.
LM Studio’s documentation says the app “can operate entirely offline.” Documents attached to a chat can also be used offline. Paid chatbot subscriptions cost about $20 per month, while the software here is free.
Licenses decide what users can do with the weights. DeepSeek’s R1 series is under the MIT License, and its model card says the models allow commercial use and modifications. Google’s Gemma 4 and OpenAI’s gpt-oss ship under Apache 2.0.
What Hardware Is Needed
Models are sized in parameters, the numbers adjusted during training. The model must fit in a graphics card’s video memory, known as VRAM, or in a Mac’s unified memory. Small models need about 8GB.
Users who want more power may need a GPU with 24GB of VRAM or more. Machines with unified memory, such as the Mac Mini or the Nvidia DGX Spark, are another option.
Quantization shrinks a model by storing each weight with fewer bits. Hugging Face describes it as using low-precision data types like 8-bit integers instead of 32-bit floating point. This cuts memory use at the cost of some accuracy.
LM Studio recommends 16GB or more of RAM on Apple Silicon Macs running macOS 14 or newer. On Windows, it recommends at least 16GB of RAM and 4GB of dedicated VRAM. Intel-based Macs are not supported.
Ollama’s library lists Gemma 4 26B at 16 to 19GB and the 31B version at 19 to 20GB. OpenAI says gpt-oss-20b needs only 16GB of memory. The gpt-oss-120b model runs within 80GB.
LM Studio has a graphical interface, and users download it from lmstudio.ai. Ollama is aimed at developers who work in a terminal. Typing “ollama pull gemma4:e2b” downloads a model, and “ollama run gemma4:e2b” opens a chat.
Ollama also opens a local API at http://localhost:11434. Model names ending in “:cloud” run in Ollama’s cloud and need no download. Ollama says prompts handled over its API are not stored or used to train models.
Google’s Multi-Token Prediction drafters for Gemma 4 are available on Ollama under Apache 2.0. They speed up generation by up to 3x using speculative decoding. LM Studio added support for Model Context Protocol servers in version 0.3.17.
The four-month average describes the most capable open models. Alibaba’s Qwen 3.8-Max has 2.4 trillion total parameters, with 95 billion active at any time. Closed models won 24 of 31 tests in Alibaba’s own comparison.
When a model spills out of VRAM into system RAM, responses can be much slower. Gemma 4’s small models offer a 128K-token context window, and the larger ones offer 256K. Decrypt’s review recommended xPrivo with a self-hosted local model for private drafting.
