How to Run Open Source LLMs Locally on a Mac

Introduction

Learning how to run open source LLMs locally on a Mac has gone from a weekend experiment to an ordinary skill for developers, analysts, and privacy-minded professionals. In March 2023, Simon Willison squeezed Facebook’s LLaMA 7B down to about 4GB with 4-bit quantization and ran it on a 64GB MacBook Pro. Today a modest laptop can host models that were out of reach for most hardware only a short time ago. The reason is that Apple Silicon pairs fast chips with a single pool of memory that the CPU and GPU share, which suits large language models unusually well. This guide explains the hardware, the file formats, the four main tools, and the settings that decide whether a model feels instant or painfully slow. You will also get a seven-step setup with real commands, benchmarks from published tests, and a candid look at where local models still fall short.

Quick Answers on How to Run Open Source LLMs Locally on a Mac

What is the easiest way to run open source LLMs locally on a Mac?

Install Ollama or LM Studio, download a quantized model that fits your unified memory, and run it locally on your Mac. Both tools handle downloads and hardware acceleration automatically.

How much memory do I need to run a local LLM on a Mac?

A 16GB Mac handles 7B to 9B models smoothly, 32GB is the realistic minimum for 30B-class models, and 70B models at 4-bit need roughly 64GB or more of unified memory.

Is MLX faster than llama.cpp on Apple Silicon?

Often for generation speed, but not always overall. On one M5 Max test, MLX could run a model at about 96 tokens per second against 70 for llama.cpp, while llama.cpp answered first.

Key Takeaways

• Unified memory lets the GPU use almost all of your RAM, so memory size, not chip speed, usually decides which models you can run.
• Four-bit quantization shrinks a model to roughly a quarter of its original size with modest quality loss, which is why 8B models fit comfortably on 16GB laptops.
• Ollama, LM Studio, llama.cpp, and MLX each suit different users, and many people install two of them rather than choosing only one.
• Running models locally keeps prompts on your machine, but it trades away some raw quality and demands attention to storage, heat, and model licenses.

Understanding How to Run Open Source LLMs Locally on a Mac

How to run open source LLMs locally on a Mac means downloading openly licensed language model weights, compressing them to fit unified memory, and running them through tools like Ollama, LM Studio, llama.cpp, or MLX so every prompt and answer stays on your own machine.

Why Apple Silicon Suits Local Language Models

Building on the basic idea of local inference, language models spend most of their time moving numbers out of memory. Arithmetic is rarely the bottleneck, so memory access speed matters more than raw processor power. Apple Silicon chips place the CPU, GPU, and memory on one package with a wide, fast connection between them. Because the GPU reads the same memory as the CPU, a model loaded once is available to both without any copying. A typical desktop graphics card, by contrast, has a small private pool of video memory and must stop to shuffle data over a slower link. That architectural difference explains why a laptop with 64GB of unified memory can load models that would need an expensive multi-card setup elsewhere.

Apple made this advantage explicit when its machine learning team released the MLX framework in December 2023. MLX keeps arrays in shared memory, so operations can run on the CPU or the GPU without moving data between them. It offers a NumPy-style Python interface and builds computation lazily, which keeps memory use predictable. The practical result is that a Mac can treat nearly all of its installed memory as model space, with the operating system and your open apps taking the remainder. Other tools that predate MLX, such as llama.cpp, reach the same hardware through Apple’s Metal graphics layer. Either route benefits from the same shared pool of memory.

Looking beyond the architecture itself, it helps to remember that newer chips keep improving the picture. Apple’s recent M5, M5 Pro, and M5 Max chips add GPU Neural Accelerators that Ollama now uses to cut the wait for the first token and raise generation speed. Older M1 through M4 machines still run local models well, and many people report excellent results on them. The gap shows up mainly in long prompts, where the initial processing step dominates the experience. If you are choosing between a newer base model and an older high-memory machine, the extra memory usually buys more capability than the newer chip does. Compare the broader chip landscape of GPUs, TPUs, and NPUs if you want context on how these designs differ.

Unified Memory and How Much of It a Model Really Needs

Turning to the arithmetic of memory, the amount a model needs comes from two parts: the weights and the working cache. A quick rule of thumb multiplies the parameter count in billions by the bytes per parameter and then adds roughly 15 to 20 percent for overhead. Sixteen-bit weights use about two bytes per parameter, eight-bit weights use one, and common four-bit formats use about half a byte. By that arithmetic, a 7B model at four bits needs around 5 to 8GB. A 70B model at four bits needs roughly 40 to 48GB, as the LMSA memory guide lays out. Those figures describe only the weights and a small buffer, which is why real usage is higher.

The second part is the key-value cache, which grows with every token of conversation you keep in context. Long context is the hidden cost, because a cache that looks trivial at a few thousand tokens can rival the model itself at 128,000. According to the Hermes Agent guide, a 9B model at 128K context needs about 16GB of cache with the default 16-bit setting. That drops to about 8GB with an 8-bit cache and about 4GB with a 4-bit cache. The same guide says a Qwen3.5-9B model in the Q4_K_M format occupies about 5.3GB on disk and needs roughly 10GB of RAM at that context with a quantized cache. Shortening the context window or quantizing the cache is therefore the cheapest way to recover memory on a small machine. It also explains why two people running the same model report very different memory use.

Finally, remember that your Mac does not give the GPU every byte. By default macOS reserves a share of memory for the system. A 16GB machine therefore cannot dedicate all 16GB to a model, and the safe budget is closer to 10 to 12GB. Advanced users can raise the GPU limit with a system setting, though the change resets on restart and leaves less room for the operating system. As a working rule, match your model size to about two-thirds of installed memory, then leave the rest for macOS, your browser, and your editor. A 16GB Mac handles 7B models smoothly, 16 to 24GB covers 13B comfortably, and 32GB is the practical minimum for 30B-class models. Plan around those memory tiers before you download anything large.

Beyond planning, watch memory pressure while a model runs, because swapping to disk turns a fast tool into a slow one. The Activity Monitor memory graph turns yellow and then red as pressure grows, which is your cue to pick a smaller model or a shorter context. Closing browser tabs and large apps before a long session frees more headroom than most people expect. Another reliable fix is choosing a mixture-of-experts model, which activates only a fraction of its parameters per token, although it still needs the full set loaded in memory. Treat memory as the budget that every other choice in this guide spends against. When in doubt, pick the smaller model and a longer context rather than the reverse.

Quantization and Model Formats: GGUF, MLX, and Safetensors

Shifting from memory to file formats, quantization is the technique that makes local models practical. It stores each weight in fewer bits, trading a small amount of precision for a large cut in size and a gain in speed. A 16-bit model squeezed to four bits shrinks to roughly a quarter of its original footprint, which is the difference between needing a workstation and fitting on a laptop. Quality loss at four bits is usually modest for chat and coding, while two- and three-bit variants degrade more noticeably. Our overview of post-training quantization for edge AI explains the underlying math in more depth. For most Mac users, a four-bit or five-bit build is the sweet spot.

Different tools expect different file formats, and mixing them up is the most common beginner mistake. GGUF is the single-file format used by llama.cpp and by Ollama’s original engine, and names like Q4_K_M describe how the weights were compressed. MLX models are distributed as folders of safetensors files with their own quantization schemes, such as four-bit or the newer mxfp4. Ollama’s recent MLX-based engine also reads safetensors models, including a Qwen3.5 build tagged with the NVFP4 format. One consequence surprises many people: every GGUF file you already pulled still runs on llama.cpp, so upgrading alone does not speed up old downloads. You must pull a model in the newer format to benefit.

The practical advice is to match the format to the tool and to keep downloads from trusted sources. Hugging Face hosts thousands of GGUF and MLX conversions, and community groups publish ready-made versions of popular models soon after release. Prefer uploads from the original model authors or from well-known conversion accounts, and check the file size against the expected figure. Safetensors files cannot run arbitrary code the way old pickle-based checkpoints could, which is one reason the community moved toward them. Keep a short note of which quantization level you downloaded so that later comparisons are fair. If a model feels dull or error-prone, try the next higher quantization before blaming the model itself.

Choosing a Model That Fits Your Mac

Stepping back from file formats to model selection, start with the job rather than the leaderboard. Everyday chat, summarization, and drafting work well on 7B to 9B models, while multi-step coding and complex reasoning reward 30B-class models or larger. A reliable starter on a 16GB machine is a recent 8B instruction-tuned model in a four-bit build, which loads in seconds and responds quickly. Mixture-of-experts models such as the 35B-A3B Qwen3.5 variant offer large-model knowledge at small-model speed, but they need more than 32GB of unified memory to load comfortably. Open families from Meta, Alibaba, Google, Mistral, and DeepSeek all publish weights you can run locally. The site’s look at Qwen3 against OpenAI and DeepSeek shows how quickly this field moves.

Next, test two or three candidates on your own prompts, because public benchmarks rarely match your daily work. A model that scores well on a leaderboard can still stumble on your formatting rules, your domain vocabulary, or your preferred answer length. Keep a small file of ten representative prompts and run each candidate through it, noting speed, accuracy, and tone. Pay attention to the license too, since some open-weight models restrict commercial use or large deployments. Delete the losers promptly, because model files are large and quickly consume a laptop drive. Over a few weeks this habit produces a short, trusted list that you can rely on without rechecking every new release.

Ollama, LM Studio, llama.cpp, and MLX Compared

Moving on from choosing a model to choosing a runner, the four main tools differ mostly in how much they hide. Ollama is a command-line tool and background service that downloads models by name and exposes a local API, which makes it the easiest path for developers. LM Studio is a graphical application that searches Hugging Face, manages downloads, and offers a chat window plus a local server. Version 0.3.4 added an open-source MLX engine and lets you load both llama.cpp and MLX models side by side. llama.cpp is the foundation underneath many tools, giving you the most control through command-line flags and a built-in server. MLX, with its language-model package mlx-lm, is Apple’s own route and shines for fine-tuning and research.

Each tool has a clear sweet spot, and the choice depends on your habits. Choose Ollama if you want one command to pull and run a model and an API that other software can call. Choose LM Studio if you prefer a visual interface, easy model discovery, and built-in structured-output controls. Choose llama.cpp when you need tight memory budgets, advanced sampling, constrained decoding, or portability across operating systems. Choose MLX when you want the fastest generation on typical four-bit models between 7B and 30B, or when you plan to fine-tune. There is no single winner, which is why many experienced users keep two engines installed and switch by task. The comparison below keeps the focus on Apple Silicon specifically.

Performance differences are real but depend heavily on the workload. In one published test on an M5 Max with 128GB, a Qwen3.5-9B model produced about 70 tokens per second under llama.cpp and about 96 under MLX. The same test showed llama.cpp reaching the first token in roughly 67 milliseconds against about 289 for MLX. A 512-token reply finished in 7.3 seconds on llama.cpp and 5.5 seconds on MLX, so generation speed won over the long run. Another analysis ran a 35B model on an M1 Max at about 8,500 tokens of context. It found llama.cpp with flash attention finished roughly nine seconds sooner, because MLX’s long prefill dominated. The lesson is that short chats favor MLX while long documents can favor llama.cpp.

Beyond raw speed, consider the features that quietly decide your daily comfort. Constrained decoding, which forces output to match a JSON schema or grammar, is a strength of llama.cpp that agents rely on. LM Studio also enforces JSON output and added prompt caching, which cut a follow-up response from about ten seconds to 0.11 seconds in its own test. MLX offers custom samplers written in Python, plus native fine-tuning, which most other runners lack. Ollama’s model library and simple naming make sharing a reproducible setup with teammates easy. Weigh these features against your real tasks instead of chasing a single benchmark number.

Speed in Practice: Prefill, Decode, and Context Length

Given the tool differences above, it is worth understanding the two phases that make up every response. Prefill is the first phase, in which the model reads your entire prompt and builds its cache. Decode is the second, in which it generates the answer one token at a time. Prefill is compute-bound and fast per token, while decode is memory-bound and therefore limited by how quickly the chip can read the weights. That is why a long pasted document produces a noticeable pause before the first word, and why the reply then streams at a steady pace. Reading speed benchmarks correctly means asking which phase each number describes.

Published figures show just how much the choice of engine matters. Ollama reported that version 0.19 with the MLX backend reached 1,851 tokens per second of prefill. Decode reached 134 tokens per second on a 35B-A3B Qwen3.5 model at int4 quantization. A separate write-up described decode speed rising from 58 to 112 tokens per second on the same machine after the switch. These gains are large, but they apply to the safetensors models that use the new engine and not to your existing GGUF library. LM Studio’s launch post cited about 250 tokens per second for Llama 3.2 1B on an M3 Max, a reminder that small models fly. Your own numbers will depend on chip, memory bandwidth, model size, quantization, and context length.

For everyday use, a few simple habits keep your responses snappy. Keep conversations short and start fresh threads when the topic changes, because every extra token of history costs time and memory. Reuse a stable system prompt so that prompt caching can skip repeated work, as explained in our guide to prompt caching and cutting LLM costs. Choose a smaller model for quick tasks such as classification or rewriting and save the large one for hard problems. Plug in the charger during long sessions, since macOS may throttle performance on battery. Finally, measure speed on your own prompts and record it, so later upgrades can be judged against evidence instead of impressions.

Beyond chat windows, every major runner can expose your model through a local web API, and that is where local models become genuinely useful. Ollama runs a background service with its own interface and an OpenAI-compatible route. LM Studio offers a local server that mimics the OpenAI API, and llama.cpp ships a server with the same style of endpoints. Because the interface matches what cloud providers use, most existing libraries and apps need only a changed base address to talk to your Mac. A model on your own laptop can therefore stand in for a paid cloud endpoint during development, testing, and offline work. The older guides to running AI locally on Windows 11 and to running your own AI chatbot locally show the same pattern on other platforms.

Once a server is running, keep it private and predictable. Bind it to the loopback address so that only programs on your own Mac can reach it, and never expose the port to a coffee-shop network. Give each project its own model name or alias so that a configuration file always points to the version you tested. Remember that a long first request is usually the model loading into memory, not a hang, and later requests run much faster. Watch the context setting, since many servers default to a short window that silently truncates long prompts. If a tool reports odd answers, check the server log first, because a wrong prompt template or truncated input is the usual culprit.

Putting Local Models to Work: Retrieval, Coding, and Everyday Tasks

In practice, the most common local workloads are document question answering, coding help, and private drafting. Retrieval lets a small model answer from your own files by fetching relevant passages and placing them in the prompt, so the model needs less built-in knowledge. A step-by-step walkthrough of a RAG chatbot built with open-source tools maps directly onto a Mac, since the embedding and generation models can both run locally. Keeping documents on the machine matters for contracts, medical notes, and unreleased research. The trade-off is that your retrieval quality depends on chunking, search, and the freshness of the index you maintain. Test with questions whose answers you already know before trusting the system with new ones.

Coding assistants are the second big use, and they stress the machine harder than chat does. Agents send long prompts full of file contents and tool results, which makes prefill speed and context length the deciding factors. Ollama says its MLX update speeds up coding agents such as Claude Code, OpenCode, and Codex when they point at a local model. For serious agent work, a Mac with more than 32GB of unified memory and a fast prefill path is the comfortable starting line. The article on a local AI coding stack challenging the cloud discusses where this approach already competes. For quick completions and refactors, a smaller model on a 16GB machine can be perfectly adequate.

Beyond those two, local models handle summarizing meeting notes, drafting emails, translating text, tagging support tickets, and brainstorming. These tasks tolerate smaller models and short contexts, so they run fast and cost nothing per request. Voice transcription, image captioning, and embedding generation also run locally with the right open models. Combine tools in a small pipeline, for instance a fast model for triage and a larger one for the hard cases. A weekly review of what you actually used the model for will show whether a bigger machine would ever pay off. Many people discover that their real workload fits comfortably within modest hardware.

Fine-Tuning and Customization on a Mac

Next, moving from using models to shaping them, a Mac can fine-tune smaller models with techniques that update only a thin layer of added weights. LoRA and its quantized cousin QLoRA train small adapter matrices while the base model stays frozen, which slashes memory needs. MLX supports LoRA, QLoRA, and DoRA training natively, and that is one of the clearest reasons to pick it over other runners. A 7B model can be adapted on a well-equipped laptop in hours using a few thousand examples. The site’s guide to fine-tuning LLMs at home with Axolotl covers the workflow on other hardware and shares the same concepts. Expect the process to be slower than on a data-center GPU, but entirely workable for experiments.

Before reaching for training, ask whether you actually need it. Prompting, retrieval, and a good system prompt solve most customization problems without touching the weights. Fine-tuning earns its place when you need a consistent style, a narrow output format, or domain vocabulary that prompts cannot reliably produce. The comparison of retrieval against fine-tuning is a useful decision guide here. If you do train, hold out a test set, keep the adapter separate from the base model, and measure the result against the untouched model. In practice, small and clean datasets generally beat large, noisy ones.

Privacy, Cost, and When Local Beats the Cloud

Turning to the strongest argument for local models, privacy is the reason many organizations care. When a model runs on your Mac, prompts and documents never leave the machine, so there is no third-party server to retain, log, or leak them. In May 2023 Samsung restricted staff use of ChatGPT and similar tools after reports that employees had uploaded sensitive source code, as TechRadar reported. The company was also said to be building its own internal AI service for employees. A local model offers a lighter version of that idea for individuals and small teams. Policy decisions about which data may reach any model belong in a written company standard.

Cost is the second argument, though it needs honest arithmetic. Local inference has no per-token fee, and heavy users can save real money once the hardware is already on their desk. The saving disappears if you buy a new high-memory Mac solely to avoid a small cloud bill. Electricity and battery wear are minor, but your time spent configuring and troubleshooting is not free. Cloud models also remain stronger on the hardest reasoning and longest-context tasks. Our explainer on how to reduce LLM inference costs helps compare the two paths.

Finally, consider the middle path, because the choice is rarely all or nothing. Many teams run sensitive or routine tasks locally and send only difficult, non-confidential requests to a cloud model. Latency also favors local use for short prompts, since there is no network round trip and no rate limit. Offline reliability is a quieter benefit on planes, in secure buildings, and during outages. This split is common enough that teams now plan for it explicitly. Decide per task, and revisit the decision as models improve.

Where Local AI Falls Short on a Mac

Stepping back from the enthusiasm, local models on a Mac have real limits that deserve plain language. The largest open models need far more memory than any laptop offers, so the best open systems often run only in compressed form or not at all. Four-bit quantization costs some accuracy, and small models make more factual and arithmetic errors than frontier cloud models. Long-context work is memory-hungry, and prefill on very long documents can feel slow even on strong chips. Apple Silicon also lacks the broad software support that NVIDIA’s CUDA ecosystem enjoys, so some research code runs only through conversions. The differences between edge AI and cloud AI explain why some of these trade-offs are unavoidable. A local model should be treated as a capable assistant for bounded tasks and not as a drop-in replacement for the strongest cloud systems.

Operational limits matter too, and they show up after the novelty fades. Model files are large, and a handful of downloads can fill a 256GB drive within a week. Sustained generation heats a thin laptop, which triggers throttling and loud fans. Software updates change behavior, as when Ollama’s switch to MLX helped new safetensors models but left older GGUF files unchanged. Each tool also has gaps, with Hannecke’s comparison noting that MLX trails llama.cpp on constrained decoding and aggressive low-bit quantization. Expect occasional rough edges, and keep a known-good setup you can fall back to.

Reliability is the last limit, because small models hallucinate and follow instructions less consistently. They may invent citations, misread tables, or ignore formatting rules after a long conversation. Retrieval, structured output, and verification can reduce those errors but never eliminate them. For anything with legal, medical, or financial consequences, treat the model’s output as a draft that a qualified person must check. Keep prompts short and explicit, and request quotes from supplied documents instead of recalled facts. Record failures you notice so that you can choose a better model or a better prompt next time.

Ethics, Licenses, and Responsible Use

Beyond the technical limits, open weights are not the same as open source in the strict sense, and licenses vary widely. Some models allow unrestricted commercial use, others cap usage by company size, and a few forbid certain applications outright. The debate over what open source AI really means is worth reading before you build on someone else’s weights, and a written data privacy standard helps too. Check the license file on the model page and keep a copy with your project. Downloading a model does not grant permission to use it for anything you like. If you redistribute a fine-tuned model, you may inherit the original restrictions.

Responsible use also means thinking carefully about outputs and about the data you feed in. A local model has no provider-side safety filters beyond what its training included, so you carry more responsibility for what it produces. Do not use it to generate deceptive content, to impersonate people, or to process other people’s personal data without a lawful basis. If you fine-tune on customer or employee data, apply the same consent and retention rules you would use elsewhere. Disclose AI assistance where your profession or publisher expects it. Treat model downloads like any software supply chain, preferring official sources and verifying file sizes and checksums when they are published.

Keeping a Local Setup Healthy: Updates, Storage, and Heat

Looking after the setup over months, the first habit is disciplined housekeeping. List your downloaded models regularly and remove any you have not used in a month. Keep models on an internal drive or a fast external SSD, since slow disks stretch load times from seconds to minutes. Note which tool and version produced a result you care about, because updates can shift speed and even output style. Back up your prompts, configuration files, and evaluation set, but not the model files, which you can download again. A clean model list also makes it much easier to compare candidates fairly.

Thermal and power management come next, and they affect speed more than most people expect. Long generations push the chip hard, and a thin laptop will throttle if its fans cannot keep up. Raise the machine on a stand, keep vents clear, and avoid running a large model on battery when you need steady speed. Sustained performance depends more on cooling than on the peak numbers printed in launch announcements. Activity Monitor shows memory pressure and GPU use, which helps you spot a runaway process. If the Mac feels sluggish, quit the model server and relaunch it, since a leaked cache can persist across many requests.

Security deserves a short routine of its own, separate from your housekeeping. Keep servers bound to the local machine, update your tools when security fixes ship, and avoid running unknown scripts that arrive bundled with a model download. Review the permissions of any app that wraps a local model, especially those that read your files. Use a separate user account or folder for experiments if you test untrusted models. Understand that prompt injection can still affect local agents that browse the web or read documents, as the site’s explainer on prompt injection attacks and defenses describes. Stay curious about new tools, but stay cautious about what you let them access.

The Future of Local AI on Apple Silicon

Looking ahead, the trend favors the Mac user, because models keep getting smaller and better while the software keeps getting faster. Ollama’s move to MLX in March 2026 shows how quickly the runner layer can improve, with its preview claiming large gains on all Apple Silicon Macs. The M5 family’s GPU Neural Accelerators add dedicated matrix hardware that mainly helps the prefill phase. Open models in the 9B to 35B range now handle tasks that once required far larger systems, and mixture-of-experts designs make big knowledge affordable. Each year the line between what needs the cloud and what runs on a laptop moves in the laptop’s favor. The site’s coverage of edge AI use cases and hardware tracks the same shift beyond Macs.

Several open questions will shape the next phase of local AI on Apple hardware. Memory prices and configurations will decide how many people can afford the 64GB and 128GB machines that unlock the largest models. Standard formats may converge, which would end the confusion between GGUF and MLX files, or they may fragment further. Better quantization could push four-bit quality closer to sixteen-bit, and techniques that compress the key-value cache will stretch context windows. Tools may also merge, with graphical apps wrapping several engines behind one interface, as LM Studio already does. Whatever happens, the habit of testing candidates on your own prompts will stay valuable.

For individuals, the practical advice is to buy memory generously, learn one tool deeply, and stay flexible about engines. Teams can pilot a local setup on existing hardware before committing to new purchases. Developers should write against the OpenAI-compatible interface so that models and runners can be swapped freely. Keep an evaluation set so each new release can be judged within an hour. The next sections turn these ideas into a concrete seven-step setup you can finish this afternoon.

How to Run Open Source LLMs Locally on a Mac: A Seven-Step Setup

Putting everything together, the following seven steps take a fresh Mac from nothing to a working local model with an API. Each step is small and testable, so you can stop at any point and still have something useful. The commands use the Terminal and Python, and you can adapt them to your own shell and editor. Pick a model sized for your memory using the earlier tiers, because a model that is too large is the usual cause of failure. Note the speed you observe after each step so that you can compare engines fairly. By the end you will have Ollama, a graphical option, a llama.cpp server, an MLX path, and a script that calls your model.

Step 1 – Check your Mac and set a memory budget

Begin by confirming which chip and how much unified memory your Mac has, because every later choice depends on those two facts. Open the Apple menu and choose About This Mac, or run the commands below in Terminal for exact numbers. Subtract about a third of the total for macOS and your open applications to get a rough model budget. A 16GB machine therefore has about 10 to 12GB for a model and its cache, and a 64GB machine has about 40GB. Write the figure down, since you will compare model sizes against it.

Next, check free disk space, because a single large model can take 5 to 40GB or more. Make sure you have at least 30GB free before starting, and more if you plan to try several models. Update macOS to a current release, because newer versions improve GPU memory handling and support the latest tools. Install the Xcode command line tools if you have not already, as Homebrew and builds depend on them. Pro tip: close heavy applications before testing so that your first benchmark reflects what the machine can really do.

# Chip and memory
sysctl -n machdep.cpu.brand_string
sysctl -n hw.memsize | awk '{printf "%.0f GBn", $1/1073741824}'

# Free disk space
df -h / | tail -1

# Command line tools and Homebrew
xcode-select --install
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2 – Install Ollama and run your first model

Ollama is the quickest way to get a model talking, so start there. Download the macOS app from the Ollama website, or install it with Homebrew, then confirm the service is running. Pull a small instruction-tuned model first, such as an 8B build, and chat with it directly in Terminal. The first run downloads roughly 5GB for an 8B four-bit model, so expect a wait that depends on your connection. Once the prompt appears, ask a few questions and note how quickly the words stream. Try at least 3 different prompts, including one long paste, so you see both prefill and decode behavior.

Ollama also reports timing statistics when you ask for them, which gives you a first speed measurement. Look at the evaluation rate, which approximates decode speed in tokens per second, and the prompt evaluation rate for prefill. If the rate is under about 10 tokens per second on a small model, check that no other heavy process is running and that the model fits in memory. You can list installed models and remove ones you no longer need. Newer Macs with more than 32GB of unified memory can also try the MLX-backed Qwen3.5 35B-A3B preview tag mentioned in Ollama’s announcement.

brew install ollama
ollama serve &

# Pull and chat with a small model, printing speed statistics
ollama run llama3.1:8b --verbose

# Housekeeping
ollama list
ollama rm llama3.1:8b

# MLX-backed preview for Macs with more than 32GB (Ollama 0.19 or later)
ollama run qwen3.5:35b-a3b-coding-nvfp4

Step 3 – Add LM Studio for a visual workflow and the MLX engine

LM Studio gives you a friendly window for finding, downloading, and testing models, which is ideal for comparison. Download the Apple Silicon build from the LM Studio website, open it, and use the search tab to find a model. Choose between GGUF and MLX builds of the same model, and download both if you have the disk space. Load each in turn and send the same prompt, noting the speed shown in the interface. The side-by-side view often settles the engine question for your own hardware. Repeat the test at 2 context lengths, such as a short chat and a long pasted article, because the winner can change.

LM Studio can also run a local server and enforce JSON-structured replies, which is useful for scripting. Its release notes describe an open-source MLX engine, KV caching across prompts, and the ability to load multiple models at once. Use the developer tab to start the server and copy the address it shows. Keep the context length modest at first, such as 8,192 tokens, since large values reserve memory up front. Pro tip: favor models labeled for instruction or chat use, because base models do not follow commands well.

# Optional: install via Homebrew instead of the website
brew install --cask lm-studio

# After starting the local server in the app, test it
curl http://localhost:1234/v1/models

Step 4 – Build a llama.cpp server for full control

llama.cpp rewards you with fine-grained control and is the engine behind many other tools. Install it with Homebrew, which provides the server program, then download a GGUF file from Hugging Face into a models folder. Start the server with full GPU offload, a context size you can afford, one parallel slot, and flash attention enabled. Quantizing the key-value cache to four bits saves several gigabytes at long context, as the Hermes Agent guide recommends for small machines. Leave the default port of 8080 in place unless something else already uses it, and keep the context near 32,768 tokens until you know your memory headroom.

Watch the startup log, which reports how many layers were offloaded and how much memory the cache needs. If the log shows layers staying on the CPU, raise the offload setting or pick a smaller model. Test the server with a simple request to the chat completions route before connecting any application. Because llama.cpp exposes many flags, save your working command in a script so you can reproduce it. Try a few cache and context settings and record the speed for each.

brew install llama.cpp
pip install -U "huggingface_hub[cli]"

# Download a quantized GGUF build (replace repo and file with your choice)
mkdir -p ~/models
huggingface-cli download  .gguf --local-dir ~/models

# Serve it: full GPU offload, 32K context, flash attention, 8-bit cache
llama-server -m ~/models/.gguf 
  -ngl 99 -c 32768 -np 1 -fa on 
  --cache-type-k q8_0 --cache-type-v q8_0 
  --host 127.0.0.1 --port 8080

Step 5 – Run an MLX model with mlx-lm

MLX is Apple’s own path and often gives the best generation speed for four-bit models between 7B and 30B. Create a Python virtual environment so that the packages stay isolated from your system. Install the mlx-lm package, which downloads converted models from Hugging Face on first use. Generate a short reply from the command line to confirm that everything works. Compare the tokens-per-second figure it prints against the numbers from Ollama and llama.cpp. Run each engine at least 3 times with the same prompt and discard the first run, which includes model loading.

The same package includes a server and Python functions, and it supports fine-tuning with LoRA. Use a four-bit community conversion of the model you tested earlier so that the comparison is fair. Expect a slightly slower first token than llama.cpp on long prompts, as the published tests showed. Fair comparisons use the same model family, the same quantization level, and the same prompt in every engine. Keep notes in a simple table, and decide which engine becomes your default. You can always keep the others installed for special cases.

python3 -m venv ~/mlx-env
source ~/mlx-env/bin/activate
pip install -U mlx-lm

# Generate with a 4-bit community conversion (downloads on first run)
mlx_lm.generate 
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit 
  --prompt "Explain unified memory in three sentences." 
  --max-tokens 200

# Optional: start an OpenAI-compatible server
mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --port 8081

Step 6 – Call your local model from code

With a server running, any script can use your model through the standard OpenAI client library. Point the base address at the local server and use any placeholder for the API key, since local servers usually ignore it. Set the model name to match what the server reports. Start with a short prompt and a modest output limit, then stream the response to see tokens as they arrive. This single pattern works with Ollama, LM Studio, llama.cpp, and mlx-lm, which is why it is worth learning. The default addresses are 11434 for Ollama, 1234 for LM Studio, 8080 for llama.cpp, and 8081 in the example below for mlx-lm.

Keep the connection details in environment variables so that switching engines needs no code changes. Add a timeout generous enough for the first request, which includes model loading. Log the time to first token and the total time for each call, which gives you your own benchmark data. Handle errors gracefully, because a server may be down or a model may not be loaded. Pro tip: write your evaluation prompts once in a file and replay them against every engine you test.

import os, time
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "  # Ollama default
    api_key="not-needed",
)
model = os.getenv("LLM_MODEL", "llama3.1:8b")

start = time.time()
first = None
text = []
stream = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Give me three tips for running LLMs on a Mac."}],
    max_tokens=300,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    if delta and first is None:
        first = time.time() - start
    text.append(delta)

print("".join(text))
print(f"time to first token: {first:.2f}s, total: {time.time() - start:.2f}s")

Step 7 – Tune memory, context, and monitoring

Finish by tuning the settings that keep your setup stable. Reduce the context window until memory pressure stays green in Activity Monitor during your longest realistic prompt. Quantize the key-value cache if your tool supports it, and pick a smaller model for background tasks. Advanced users on a Mac with plenty of memory can raise the GPU memory limit, but the change resets on restart and should leave several gigabytes for macOS. Record your final configuration so that you can rebuild it quickly after an update. A 64GB Mac with the limit raised to 48GB, for instance, leaves 16GB for macOS and your other applications.

Then set up simple monitoring so that problems show up early. Check memory pressure and temperature during a long generation, and note any throttling. Keep your ten-prompt evaluation file and rerun it after each tool or model update. A five-minute regression check after every upgrade prevents the slow drift that makes local setups feel unreliable. Delete models you no longer use, and review your tool versions monthly. With these habits in place, your Mac becomes a dependable private AI workstation.

# Watch memory pressure and swap while a model runs
memory_pressure | head -5
sysctl vm.swapusage

# Show the current GPU memory limit (0 means the macOS default)
sysctl iogpu.wired_limit_mb

# Advanced: raise the limit to 48 GB on a 64 GB Mac (resets on reboot)
sudo sysctl iogpu.wired_limit_mb=49152

Key Insights

• Ollama’s MLX preview reported 1,851 tokens per second of prefill on a 35B-A3B Qwen3.5 model, which shows how much the runner layer can change the speed of a fixed machine.
• That same Ollama announcement says the faster path needs a Mac with more than 32GB of unified memory, so memory capacity still decides who can use the newest speedups.
• A 9B model at 128K context needs about 16GB of cache at the default setting, and the Hermes Agent guide shows a 4-bit cache cuts that to about 4GB.
• In an M5 Max test, MLX generated about 96 tokens per second against 70 for llama.cpp, yet the same published benchmark shows llama.cpp reaching the first token four times sooner.
• A 70B model at 4-bit needs roughly 40 to 48GB of memory, according to the LMSA sizing guide, which rules out most 16GB laptops without heavy compromises.
• LM Studio reported that KV caching across prompts cut a follow-up response from about 10 seconds to 0.11 seconds, so reusing context is worth more than a faster chip.
• Decode speed on one Mac reportedly rose from 58 to 112 tokens per second after Ollama adopted MLX, but GGUF files you already own saw no gain.

Taken together, these figures describe a field in which software choices matter almost as much as hardware. Memory capacity sets the ceiling on model size, while the engine and cache settings determine how close you get to the machine’s real speed. Published benchmarks disagree about which engine wins because prompt length, model family, and quantization all change the answer. The most reliable practice is therefore to measure your own prompts on your own Mac. Teams that keep a small evaluation file can adopt new releases with confidence and discard hype quickly. The sensible answer to how to run open source LLMs locally on a Mac is to size memory first, then test engines, then tune context.

Tool	Best For	Interface	Main Strength	Main Limitation
Ollama	Developers wanting one-command setup	Command line and local API	Simple model pulls and a ready API	Fewer low-level controls than llama.cpp
LM Studio	Visual model browsing and comparison	Desktop application	Runs llama.cpp and MLX models side by side	Closed-source application shell
llama.cpp	Maximum control and portability	Command line and server	Constrained decoding and tight memory budgets	Many flags to learn
MLX and mlx-lm	Fast four-bit generation and fine-tuning	Python and command line	Native LoRA training and custom samplers	Slower first token on long prompts
Ollama with MLX preview	Large-memory Macs seeking speed	Command line and local API	Reported 134 tokens per second decode	Needs more than 32GB and safetensors models
GGUF models	Wide tool compatibility	File format	Single portable file	Not read by the MLX engine
Safetensors MLX models	Apple-native pipelines	File format	Matches the MLX framework	Format split adds confusion

Real-World Mac Setups Running Open Models Locally

Simon Willison’s LLaMA on a MacBook Pro

Turning to documented setups, developer Simon Willison ran Facebook’s LLaMA 7B model on a 64GB M2 MacBook Pro in March 2023. He used llama.cpp, a C and C++ port by Georgi Gerganov. The key step was a script that quantized the model to 4 bits, which brought the weights down to about 4GB. That is a reduction of roughly 70 percent from the 14GB that 16-bit weights would need, by simple arithmetic. Willison had long wanted to run a GPT-3-class model on his own hardware, and his short write-up helped popularize the approach among developers. The post reported no speed figures, so readers had to measure performance on their own machines. The setup still required a generous 64GB of memory, which most buyers of the era did not own.

LM Studio’s MLX Engine on an M3 Max

Among graphical tools, LM Studio released version 0.3.4 on October 8, 2024, adding an open-source MLX engine for running models efficiently on Apple Silicon. The release let users load llama.cpp and MLX models side by side and enforce JSON output through the interface. In its own tests, the LM Studio team reported about 250 tokens per second for Llama 3.2 1B on an M3 Max. A second test used a roughly 3,000-token article with an 8B four-bit model. KV caching cut a repeat response from about 10 seconds to 0.11 seconds, a reduction of roughly 99 percent. Those numbers made a strong case for caching, since the saving appeared on ordinary chat-style workloads. The 1B speed does not transfer to larger models, and the engine still requires an Apple Silicon Mac, so Intel owners were left out.

Ollama’s MLX Preview on Large-Memory Macs

Looking at command-line users, Ollama announced a preview of version 0.19 on March 30, 2026, after it adopted Apple’s MLX framework for its Apple Silicon path. The company reported 1,851 tokens per second of prefill and 134 tokens per second of decode with int4 quantization on a Qwen3.5 35B-A3B coding model. Macs with the M5, M5 Pro, and M5 Max chips gained extra speed from the GPU Neural Accelerators, improving both the first-token wait and the generation rate. Ollama’s announcement listed that single model as the preview’s supported option, with more to follow. It also recommended a Mac with more than 32GB of unified memory, which excludes many popular laptops. A separate report described decode speed rising from 58 to 112 tokens per second, an increase of roughly 93 percent, though GGUF files still ran on the old engine. Coverage by 9to5Mac noted that many users might not meet that requirement.

Lessons From Local AI Rollouts and Engine Choices

Case Study: Samsung’s Restrictions on Cloud Chatbots

Turning to corporate lessons, Samsung faced a data-handling problem in early 2023 when employees reportedly pasted sensitive material into ChatGPT, including internal source code. The company responded in May 2023 by restricting generative AI tools on company phones, tablets, and computers, and it warned that violations could lead to disciplinary action. Samsung was also reported to have built, or to be building, its own internal AI service for employees, with prompts limited to 1,024 bytes. That cap was a deliberate reduction in what any single prompt could expose. TechRadar’s coverage tied the ban to three reported leak incidents, which showed how quickly a convenient cloud tool can become a compliance risk. The episode became a standard argument for keeping confidential prompts on infrastructure the company controls. Bringing a model in-house removes the third-party retention concern, though it does not remove the need for access controls and logging.

The limitations of this story are worth stating plainly before drawing conclusions. Several details of the story come from reports rather than a public Samsung statement, and the article does not describe exactly what each incident exposed. A 1,024-byte prompt cap would also make serious coding help impractical, which suggests the first internal tools were deliberately constrained. A local model on a developer’s Mac avoids the upload problem entirely, but it brings its own quality gap against frontier systems. Organizations still need policy on which data may touch any model, local or remote. The lesson is that private inference is a control, not a complete governance plan.

Case Study: Choosing Between llama.cpp and MLX on a Mac

Next, an engineer comparing runners found a common problem: published benchmarks seemed to contradict each other, and the choice between llama.cpp and MLX looked like a coin flip. The author analyzed the two engines in a detailed comparison, noting that llama.cpp is a full deployment stack while MLX is a numerical framework with a language-model wrapper. On an M1 Max running a 35B model at about 8,500 tokens of context, MLX’s effective speed dropped sharply once prefill was counted. llama.cpp with flash attention finished the same response about nine seconds sooner, a reduction in total response time. For typical four-bit models between 7B and 30B, the author still found MLX the better fit, especially for fine-tuning and native Swift apps. The author developed a recommendation to run both engines behind a thin abstraction layer so the default can change later.

The analysis has limits that a careful reader should keep in mind. It rests largely on a single machine and a few models, and results shift with each software release. Constrained decoding, aggressive low-bit quantization, and long-context flash attention were named as areas where llama.cpp still leads. A team following this advice would need to maintain two code paths, which adds engineering overhead. The approach still beat committing to one engine early, because it let the team measure rather than guess. Treat any benchmark, including this one, as a starting point for your own tests.

Case Study: Apple’s MLX Framework Launch

Finally, Apple’s machine learning team faced a gap in late 2023, because most array frameworks assumed separate CPU and GPU memory and wasted effort copying data on Apple Silicon. Its answer was MLX, released in December 2023 as an open framework with a NumPy-style Python interface and a C++ API. Computerworld’s report explained that arrays live in shared memory, so operations can run on the CPU or GPU without transfers. MLX also offers lazy computation, automatic differentiation, and dynamic graph construction that avoids slow recompilation when shapes change. Within a year, tools such as LM Studio had built MLX engines. In 2026 Ollama moved its Apple Silicon path onto the framework, with a reported 93 percent increase in decode speed. That adoption shows how a single low-level design choice can reshape an entire software ecosystem.

The launch also came with several caveats that users should know. Apple aimed the framework at researchers and developers, so it was not a consumer product, and the first release supported only the CPU and GPU. The Neural Engine was still only a future possibility, and the initial coverage published no performance benchmarks. The ecosystem also split in two, with GGUF models serving llama.cpp and safetensors models serving MLX. Users must now track which format each tool expects, and a download that works in one app may be useless in another. The framework’s value is real, but it arrived with extra complexity for everyday users.

Common Questions About Running Open Source LLMs Locally on a Mac