Local Model Deployment

Many people think AI can only be used in the cloud: when using ChatGPT, data is sent to OpenAI's servers, and when using Claude, requests are sent to Anthropic's data centers.

In fact,AI models can run entirely on your own computer。

Imagine: you can use AI without a network, sensitive data never leaves your computer, no need to pay per call, and you can customize it however you want.

This is the value of local model deployment.

This module will take you from zero to running your first local AI model on your own computer.

Why Local AI Is Needed

Cloud AI is convenient, but local AI has irreplaceable advantages.

Data Privacy Protection

This is the primary reason many people choose local deployment.

If you are dealing with contracts, medical records, financial data, or internal documents, sending this data to third-party servers is risky.

When running locally,all data is processed on your own computer and never leaves your device。

Offline Environment Usage

On airplanes, trains, or in places with unstable networks, cloud AI is unusable.

Once downloaded, local models are available forever and do not require a network connection.

Cost Control

Cloud AI charges based on the number of calls or tokens, and heavy usage can become a significant expense.

Local models are downloaded once and can be used freely afterward, with no additional costs.

Customization Possibilities

You cannot modify cloud models; you can only use the features provided.

With local models, you can fine-tune, quantize, modify prompt templates, and customize them entirely according to your own needs.

Comparison of these advantages:

Not that cloud AI is bad, but ratherBoth have their own applicable scenariosFor daily chat, use the cloud; for sensitive data, use local; for simple tasks, use the cloud; for special customization, use local.

Hardware Requirements Assessment

Whether a local model can run and how fast it runs mainly depends on your hardware configuration.

CPU Inference vs GPU Inference

There are two ways to run models: CPU inference and GPU inference.

CPU inference uses the computer's main processor to run models. The advantage is good compatibility; almost all computers can run it. The disadvantage is slow speed; large models may take a long time.

GPU inference uses graphics cards. The core advantage isStrong parallel computing capability, making model runs dozens of times faster。

For NVIDIA graphics cards, you need to install the CUDA toolkit; for AMD graphics cards, you can use ROCm; for Intel graphics cards, you can use OpenVINO.

Relationship Between VRAM Requirements and Model Size

VRAM determines how large a model you can run.

The more parameters a model has, the more VRAM it requires. However, with quantization techniques, you can run larger models with less VRAM.

Advantages of the Mac M Series

Apple Silicon (M1, M2, M3 series chips) has unique advantages in local AI.

Apple's Metal framework and unified memory architecture make Macs highly efficient at running local models.

Unified memory meansCPU and GPU share memoryand can automatically borrow memory when VRAM is insufficient.

A MacBook Pro with 16GB of unified memory can smoothly run quantized 7B and 13B models.

Recommended Configuration Plans

Based on different budgets and needs, here are several configuration options:

Don't be intimidated by the term "large model." Today's quantization technology already allows 7B models to run smoothly on ordinary computers, and the capability of 7B models is sufficient for most daily tasks.

Ollama: The Easiest Local AI Tool

• Ollama official website:https://ollama.com/
• Ollama supported models list:https://ollama.com/search
• Ollama tutorial:https://www.example.com/ollama/ollama-tutorial.html

Ollama is currently the most popular local model tool. To summarize in one sentence:Install models like installing apps, use AI like using the command line。

Installation and Configuration

Ollama supports macOS, Linux, and Windows, and the installation process is very simple.

Installer download link:https://ollama.com/download

Install with one command:

curl -fsSL https://ollama.com/install.sh | sh

macOS users can download the installer directly, or install with Homebrew:

# macOS 使用 Homebrew 安装
brew install ollama

# 或者下载安装包：https://ollama.com/download

Windows users can simply download the installer from the official website and run it.

After installation, start the Ollama service:

ollama serve

Once you see the service started successfully, you can start using it.

Downloading Models

Ollama's model library is very rich, including mainstream models such as Llama, Qwen, Gemma, and Mistral.

Downloading a model takes just one command:

You can also useollama run + model namecommand, which will automatically download the specified model if it is not already present:

ollama run qwen

Run the qwen model, downloading it if it's not available.

Each model has different versions, such as 8B, 70B, or different quantized versions.

You can useollama listto view already downloaded models:

ollama list

Output similar to:

NAME              ID              SIZE      MODIFIED
qwen3.5:latest    6488c96fa5fa    6.6 GB    3 months ago

Command Line Usage

After the model is downloaded, you can run it directly to start a conversation:

# 运行 Llama 3，没有会下载
ollama run llama3

# 运行 Qwen，没有会下载
ollama run qwen3.5

qwen3.5 is more than sufficient for ordinary tasks:

Then you can talk directly to the model:

>>> 你好，请介绍一下你自己
你好！我是 Llama 3，由 Meta 开发的 AI 助手。我可以帮你回答问题、
写作、编程、分析数据等。有什么我可以帮你的吗？

>>> 用 Python 写一个 Hello World 程序
好的，这是一个简单的 Python Hello World 程序：

print("Hello, World!")

>>> /bye  # 输入 /bye 退出

Common commands:

API Interface Calls

Ollama comes with an HTTP API, so you can call it with code, not just from the command line.

By default, Ollama is athttp://localhost:11434to provide API service.

Call Ollama API with Python:

Run this script:

If you want a higher-level wrapper, you can use Ollama's official Python library:

Integration with Open WebUI

The command line is convenient, but a graphical interface is more friendly.

Open WebUI is a very beautiful web interface that can connect to Ollama, allowing you to use local models just like ChatGPT.

Install Open WebUI:

After installation, open in browserhttp://localhost:3000, and you'll see an interface that looks a lot like ChatGPT.

Ollama + Open WebUI is currently the most recommended local AI combo—simple to install, friendly interface, and powerful, suitable for the vast majority of users.

LM Studio: Graphical Interface Operations

If you don't like the command line, LM Studio is a great choice—Pure graphical interface, downloading and running models are all done in the window.。

Installing LM Studio

LM Studio supports Windows, macOS, and Linux.

From the official websitehttps://lmstudio.aiDownload the installation package and just install it directly.

Model Download and Management

Open LM Studio, the model library is on the left.

You can search by model name, for example search for "llama", "qwen", "mistral", find the model you want, and click download.

LM Studio will automatically list different versions of the same model:

Downloaded models will appear in the "My Models" list.

Local API Service

LM Studio can also provide an API service, compatible with OpenAI's API format.

This means that for the OpenAI code you write, you only need to change the API address and Key to use a local model.

This feature is very useful—you can use the same code to switch between cloud models and local models as needed.

For users new to local AI, I recommend: first get started with LM Studio to understand the basic concepts and operations; once familiar, use Ollama for automation and integration.

Getting Started with Model Quantization

Quantization is a key technology that allows large models to run on ordinary computers—Run larger models with less VRAM while maintaining most capabilities。

What Is Quantization

Simply put, quantization is reducing the precision of model parameters.

Native models typically store each parameter using 16-bit floating point (FP16) or 32-bit floating point (FP32).

After quantization, we use 8-bit integers (INT8) or even 4-bit integers (INT4) for storage.

Precision decreases, but the size and VRAM requirements are greatly reduced.

For example:

The rule of thumb: Q8 quantization has almost imperceptible quality loss, and Q4 quantization is sufficient for most tasks.

GGUF Format Explained

GGUF is currently the most popular quantized model format.

GGUF's predecessor is GGML, developed by the llama.cpp project.

Advantages of this format:

• First,Good cross-platform compatibility—Works on Windows, macOS, and Linux; runs on both CPU and GPU.

• Second,Supports multiple quantization levels—From FP16 to Q3, choose as needed.

• Third,Fast inference speed—Heavily optimized for CPU and GPU.

Most local model tools (Ollama, LM Studio, llama.cpp) now support GGUF format.

Differences Between Q4/Q8 Quantization

Q4 and Q8 are the two most commonly used quantization levels.

• Q8 quantization is 8-bit; each parameter is stored as an 8-bit integer. The advantage of Q8 isMinimal quality loss, almost identical to the original model; the disadvantage is that the size is still not small enough.

• Q4 quantization is 4-bit; each parameter is stored as a 4-bit integer. The advantage of Q4 isSmall size and fast speed; the disadvantage is that quality drops noticeably on complex tasks.

How to choose:

AWQ/GPTQ Quantization Schemes

In addition to GGUF quantization, there are two other common quantization schemes: AWQ and GPTQ.

AWQ (Activation-aware Weight Quantization) is characterized byRetaining weights that have a large impact on the results, resulting in smaller quality loss after quantization.

GPTQ (Gradient-based Post-Training Quantization) is characterized byUsing gradient information to optimize quantization, suitable for scenarios with GPU.

These two schemes are mainly used on NVIDIA GPUs that support CUDA.

Simple comparison:

For beginners, don't worry about these technical details—just use the Q4_K_M version recommended in Ollama or LM Studio. It is the best balance point verified by extensive testing.

Model Selection Guide

There are hundreds of open-source models now; it's important to choose one that suits your needs.

Comparing 7B/13B/70B Models

The number of parameters is an important metric—7B, 13B, and 70B are the most common specifications.

For most people, the sweet spot is the 13B Q4 quantized version—good capability with acceptable speed.

Recommended Models with Good Chinese Support

Many foreign models have mediocre Chinese capabilities. Here are a few Chinese-friendly models:

For Chinese tasks, Qwen is the first choice; its Chinese understanding and generation are top-tier among open-source models.

Task-Specific Model Selection

Different models have different strengths; choose based on your task:

A bigger model isn't always better, nor is a newer one—one that fits your hardware and your taskis the best. Start by trying a few 7B models to get a feel, then upgrade.

Hands-On: Building a Fully Offline Local Knowledge Base

One of the most practical use cases for local models is building your own private knowledge base—feed local documents to the model and have it answer questions based on them.。

The entire system runs fully offline; data never leaves your computer.

Technical Approach

We use RAG (Retrieval-Augmented Generation) technology:

1. Split documents into chunks

2. Use an embedding model to convert each chunk into a vector

3. Store vectors in a vector database

4. When asking, convert the question into a vector too

5. Find the most relevant document chunks in the vector database

6. Send the relevant documents and the question together to the local LLM

7. The LLM answers based on the document content

This sounds complex, but it's simple with existing tools.

Implementation with Ollama + LangChain

We'll use Python to implement a simple version:

Before using this system, you need to download the embedding model:

This way, you have a private knowledge base system that runs entirely locally.

You can add your own notes, documents, and e-books, and let AI help you search and answer questions.

All data in this system stays on your local machine—documents on your hard drive, vector database in your directory, models on your computer—so you don't need to worry about data leaks at all.

Performance Optimization Tips

The local model is running, but how can you make it faster? Here are some practical tips.

Choosing the Right Quantization Level

This is the simplest and most effective optimization—the lower the quantization level, the faster the speed.

If Q4 works, don't use Q8; if Q8 works, don't use FP16.

Adjusting Context Window Size

The larger the context window, the more computation is required.

If you don't need a long context, you can set it in Ollama's Modelfile:

Reducing the context from the default 8192 to 2048 will make it noticeably faster.

Using GPU Acceleration

If you have an NVIDIA GPU or an Apple Silicon Mac, make sure Ollama is using the GPU rather than the CPU.

Ollama will automatically detect and use the GPU, but sometimes you need to verify it.

In the Ollama startup logs, you should see messages like "Metal GPU activated" or "CUDA activated".

Using Faster Sampling Parameters

Some sampling parameters can affect speed:

Parameter descriptions:

Disabling Unnecessary Features

If you're using API calls and don't need streaming output, you can set stream to false.

Although this won't make inference faster, it can reduce network transmission overhead.

Performance optimization is an art of balance—speed, quality, and memory cannot all be achieved at once. Find the balance point that suits your needs.