Run AI Locally: Complete Setup Guide
In This Guide
What Running AI Locally Actually Means
When you use ChatGPT, Claude, or Gemini through their websites, your prompts travel over the internet to massive data centers where thousands of GPUs process your request. The response travels back the same way. You are renting compute time on hardware owned by someone else, and every word you type passes through their servers.
Running AI locally flips this model entirely. You download a language model file onto your own computer, install a lightweight inference engine to run it, and interact with the model right on your machine. The model loads into your RAM and GPU memory, processes your prompts using your own processor, and generates responses without any network connection at all. You could disconnect your ethernet cable and Wi-Fi, and the model would keep working exactly the same way.
The language models available for local use are open-source or open-weight models released by organizations like Meta (Llama), Alibaba (Qwen), Mistral AI, Google (Gemma), and Microsoft (Phi). These models come in different sizes measured by parameter count, typically ranging from 1 billion to 70 billion parameters. Smaller models run on modest hardware, while larger models need serious GPU memory. The key breakthrough in recent years has been quantization, a compression technique that shrinks model files by 50 to 75 percent while preserving most of the output quality, making models that once required enterprise hardware practical on consumer machines.
In practical terms, a 7 to 8 billion parameter model quantized to 4-bit precision needs roughly 5 to 6 GB of memory and runs comfortably on most computers sold in the last three years. That same model at full precision would require 16 GB. Quantization is what made local AI viable for ordinary users, and tools like Ollama handle it automatically so you never need to think about the technical details.
Why People Run AI on Their Own Machines
Privacy is the most compelling reason. When you run a model locally, your data never leaves your computer. There is no server log, no usage analytics, no terms of service granting the provider rights to your inputs. For developers working with proprietary code, lawyers reviewing confidential documents, medical professionals handling patient data, or anyone who simply values their privacy, local AI eliminates the trust problem entirely. You do not need to read a privacy policy or hope a company keeps its promises. The data physically cannot leave your machine because the model has no network connection.
Cost is another major factor. Cloud AI services charge per token, which means every prompt and every response costs money. For individual use, this might be $20 to $200 per month depending on usage. For a team of ten developers, annual costs can reach $6,000 to $24,000. A local setup has a one-time hardware cost, and after that, every query is free. If you process thousands of documents, run continuous inference, or simply use AI heavily throughout your workday, local deployment becomes dramatically cheaper over time.
Speed and availability matter too. Local inference has no network latency, no rate limits, no outages, and no waiting in queues during peak hours. Your model responds as fast as your hardware can generate tokens, and it is always available. There is no service status page to check when you need to get work done.
Customization gives local users capabilities that cloud services cannot match. You can fine-tune models on your own data, create custom system prompts without character limits, run multiple models simultaneously for different tasks, and integrate AI into your workflow in ways that API rate limits and pricing tiers would make impractical through cloud services.
Finally, there is the learning value. Running models locally teaches you how AI actually works, from model architectures and quantization formats to inference optimization and prompt engineering. This hands-on understanding is valuable whether you are a developer, a researcher, or simply someone who wants to understand the technology shaping the world.
Hardware You Need
The hardware question comes down to three components: RAM, GPU (or lack of one), and storage. Your processor matters less than you might expect, since modern CPUs are fast enough that they rarely bottleneck inference.
RAM
System RAM determines how large a model you can load. A quantized 7B model needs about 5 to 6 GB of RAM during operation, so 8 GB of total system RAM is the bare minimum (your operating system and other applications need the rest). For comfortable use with room for a web browser and other tools, 16 GB is the recommended starting point. At 16 GB, you can run 7B to 13B parameter models smoothly. With 32 GB, you open the door to 30B+ parameter models. Running 70B models typically requires 48 to 64 GB of RAM.
GPU
A dedicated GPU with sufficient VRAM dramatically accelerates inference. An NVIDIA RTX 3060 with 12 GB of VRAM, a card that costs around $250 to $300 used, can run 7 to 8B parameter models at 30 to 60 tokens per second. Without a GPU, the same model runs at 3 to 15 tokens per second on CPU alone, which is usable but noticeably slower.
The GPU sweet spot for most users is 8 to 12 GB of VRAM, which handles the popular 7 to 8B models with excellent speed. Cards with 16 GB of VRAM step up to 13 to 14B parameter models. If you want to run 70B class models entirely in GPU memory, you need 40 to 48 GB of VRAM, which means professional-grade cards like the NVIDIA A6000 or multiple consumer GPUs.
Apple Silicon Macs deserve special mention because their unified memory architecture lets the GPU access all system RAM, not just dedicated VRAM. A Mac Mini M4 with 32 GB of unified memory can run 30B+ models that would require an expensive dedicated GPU on a Windows or Linux machine. Mac performance per dollar for local AI inference is exceptionally competitive, generating roughly 12 to 25 tokens per second on models that would be out of reach for similarly priced PC configurations.
Storage
Model files range from 2 GB for small quantized models to 40+ GB for large ones. An SSD is strongly recommended since loading models from a mechanical hard drive can take minutes instead of seconds. Plan for at least 50 GB of free SSD space if you want to keep several models downloaded and ready to use.
Choosing the Right Model
The open-source model landscape has matured significantly. In mid-2026, several model families stand out for local use, each with different strengths.
General Purpose
Qwen 3 from Alibaba is the current recommendation for most local users. It offers strong reasoning, solid coding ability, support for over 100 languages, and ships under the Apache 2.0 license with no commercial restrictions. Available in sizes from 0.6B to 235B parameters, there is a Qwen 3 variant for virtually every hardware configuration.
Llama 3.3 from Meta remains a strong choice, particularly the 8B and 70B variants. It excels at instruction following and has the largest ecosystem of fine-tuned variants for specialized tasks. The Llama community has produced thousands of specialized fine-tunes for coding, creative writing, roleplay, medical knowledge, legal analysis, and countless other domains.
Coding
For code generation and programming assistance, Qwen 3 Coder and DeepSeek Coder V2 lead the pack at models you can run locally. GLM-4 from Zhipu AI has also shown exceptional coding benchmark results. These models understand dozens of programming languages and can generate, explain, debug, and refactor code with accuracy that approaches cloud-based alternatives.
Small and Fast
For machines with limited RAM, Phi-4 Mini from Microsoft and Gemma 3 from Google pack surprising capability into packages that run on 4 to 8 GB of RAM. These 1B to 4B parameter models handle basic question answering, summarization, and simple coding tasks at very high speeds, often exceeding 50 tokens per second even on CPU.
Reasoning
For complex reasoning tasks, QwQ 32B (also from Alibaba) and DeepSeek R1 offer chain-of-thought reasoning capabilities similar to cloud-based reasoning models. These models think through problems step by step and show their work, which is valuable for math, logic, analysis, and planning tasks. They require more RAM (32 GB minimum recommended) but produce noticeably better results on challenging problems.
Core Tools: Ollama and Open WebUI
Ollama
Ollama is the most popular tool for running language models locally. It handles downloading models, managing quantization formats, allocating GPU and CPU resources, and serving models through a local API, all through simple command-line instructions. Installing Ollama takes one command on Mac and Linux, or a single installer download on Windows. Running a model is equally simple: type ollama run llama3.3 and Ollama downloads the model (if needed) and starts an interactive chat session.
Behind the scenes, Ollama automatically detects your GPU, loads as much of the model into VRAM as possible, spills the rest into system RAM, and optimizes the inference settings for your hardware. It supports NVIDIA GPUs, AMD GPUs, Apple Silicon, and CPU-only operation. You do not need to configure anything. The model library includes hundreds of pre-quantized models ready for one-command installation.
Ollama also runs a local API server on port 11434, which means any application that can make HTTP requests can talk to your local models. This API is compatible with the OpenAI format, so many existing tools and scripts that were built for ChatGPT work with Ollama models with minimal changes.
Open WebUI
Open WebUI gives you a browser-based chat interface for your local models that looks and feels like ChatGPT. It connects to Ollama (or other backends like LM Studio) and adds features that the command line lacks: conversation history, model switching without restarting, file uploads for document analysis, multi-user accounts with separate chat histories, and a clean visual interface that makes local AI feel polished and professional.
Open WebUI runs as a Docker container or a Python application, connecting to your Ollama instance over the local network. Once set up, you open a browser tab and start chatting, switching between models with a dropdown menu. It is completely free, open-source, and can be shared with other people on your local network.
Getting Started in Three Steps
The fastest path from zero to running AI locally involves three components: Ollama for model management, a model download, and optionally Open WebUI for a visual interface.
Step 1: Install Ollama. On Mac or Linux, run the one-line installer from the Ollama website. On Windows, download and run the installer. The installation adds a background service that manages models and serves the API.
Step 2: Download and run a model. Open a terminal and type ollama run qwen3:8b for a strong general-purpose model, or ollama run llama3.3 for the Meta alternative. Ollama downloads the model (typically 4 to 5 GB for an 8B model) and drops you into an interactive chat. You are now running AI locally.
Step 3: Add Open WebUI (optional). If you want a browser interface, install Docker and run the Open WebUI container with a single command. It automatically detects your Ollama instance and presents all your downloaded models in a familiar chat interface. This step is optional since the Ollama command-line interface works perfectly for many users.
The entire process, from nothing installed to chatting with a local model, typically involves downloading two things (Ollama and a model) and typing two commands. There is no account creation, no API key, no billing setup, and no terms of service to accept.
Performance Expectations
Understanding what to expect from local AI prevents disappointment and helps you choose the right model for your hardware.
Speed
Token generation speed depends primarily on whether the model fits in your GPU memory. A 7B model running entirely on a modern GPU generates 30 to 60 tokens per second, which feels instantaneous and is faster than most people can read. The same model on CPU generates 5 to 15 tokens per second, which is clearly slower but still usable for most tasks. For reference, average human reading speed is about 4 to 5 words per second, so even CPU-only inference is often fast enough to read as it generates.
Larger models are proportionally slower. A 70B model on a high-end GPU might generate 15 to 25 tokens per second, while on CPU it could drop to 1 to 3 tokens per second. This is where hardware investment matters most.
Quality
Local models have closed the gap with cloud services significantly. For most everyday tasks, including writing assistance, coding help, question answering, summarization, and brainstorming, an 8B parameter model produces results that are genuinely useful and often indistinguishable from cloud alternatives. The gap widens on complex multi-step reasoning, very long context handling, and tasks requiring broad world knowledge, where the largest cloud models still have an edge.
The practical difference matters less than the benchmark numbers suggest. A local 8B model that is always available, free, private, and fast often delivers more value than a cloud model that costs money, requires internet, and has rate limits, even if the cloud model scores higher on benchmarks.
What Local AI Handles Well
Code generation and review, text summarization, question answering from provided context, translation, rewriting and editing, data extraction, brainstorming, and general conversation all work excellently on local models. Tasks that involve processing your own documents or codebases are particularly well-suited to local AI since you can feed the model large amounts of private data without privacy concerns.
Where Cloud Still Leads
Tasks requiring very large context windows (100,000+ tokens), cutting-edge reasoning on novel problems, real-time web access, multimodal understanding (images, audio, video), and specialized enterprise features like function calling with complex tool chains still favor cloud services. These gaps are narrowing with each new open-source model release, but they exist today.
When Local Beats Cloud (and When It Does Not)
The decision between local and cloud AI is not binary. Many users run both, using local models for private or high-volume tasks and cloud models for tasks that require maximum capability.
Choose local when: privacy matters, you process sensitive data, you want unlimited free usage, you need offline access, you want to learn how AI works hands-on, or your usage volume makes per-token pricing expensive.
Choose cloud when: you need the absolute best model quality regardless of cost, you lack suitable hardware, you need multimodal capabilities, you work with extremely long documents that exceed local context limits, or you need features like web browsing and code execution that cloud platforms build in.
Choose both when: you want privacy for sensitive work but maximum quality for complex tasks. Many developers use local models for code completion, documentation writing, and data processing while keeping a cloud subscription for architectural decisions, complex debugging, and tasks where the extra quality justifies the cost.
Going Further
This guide covers the fundamentals. The articles below dive deep into every aspect of running AI locally, from hardware selection and model comparison to step-by-step installation on every major platform. Whether you are evaluating whether local AI is right for you or ready to build a complete self-hosted AI workstation, the resources below will guide you through every step.