Set Up Your Own Local LLM in 5 Steps with Ollama

Last month, during a client project, while discussing the risks of sending sensitive data to external LLM APIs for a reporting tool, the need to re-evaluate local LLM solutions arose. Specifically, privacy concerns and the anticipation of exponentially increasing API costs in the coming period triggered this search. In this process, Ollama, which made the setup incredibly easy, offers a very practical solution for anyone who wants to run a local Large Language Model (LLM) on their own server or even a powerful workstation.

In this post, I will explain how you can set up your own local LLM using Ollama in five simple steps, and share my experiences during this process. Setting up a local LLM is an important way to protect data privacy, reduce costs, and gain more control over the model.

Why Do We Need Local LLMs?

Local LLMs offer distinct advantages compared to cloud-based APIs and become indispensable in certain scenarios. For a side product I’ve been working on that processes sensitive financial data, reducing dependency on cloud services and maximizing data privacy was a critical priority for me. In such applications, keeping user data under their own control is of great importance for both legal compliance and customer trust.

The conveniences offered by cloud APIs cannot be overlooked, but as data processing volume increases, costs rise rapidly. Especially in repetitive and large-scale text analyses or ERP integrations like production planning, every API call has a cost. By setting up a local LLM, you can significantly reduce these costs and achieve a much more sustainable solution in the long run. Furthermore, the ability to work independently of an internet connection provides a critical advantage for uninterrupted operations. When I think back to operations that halted due to network outages, this independence is not to be underestimated.

Local LLMs also offer greater control and customization possibilities over models. When you want to fine-tune with your own data or use models optimized for a specific task, local setups give you this flexibility. For example, when I developed a custom language model for operator screens in a manufacturing ERP, the model needed to understand factory jargon and workflows. For such niche uses, general cloud models might not provide sufficient performance, whereas fine-tuning on a local model can yield much more accurate results. This control allows you to directly manage not only the model’s behavior but also its performance and resource consumption.

What is Ollama and Why Did I Choose It?

Ollama is an open-source platform that simplifies running Large Language Models (LLMs) on local machines. It streamlines the installation, model download, and execution processes, enabling any user or developer with a GPU to experience LLMs on their own device. I’ve been experimenting with various LLMs for a while, developing multi-provider fallback strategies using different providers like Groq, Cerebras, and OpenRouter. However, the latencies and data privacy concerns introduced by these external dependencies pushed me towards local solutions. Ollama makes this complex process manageable with a single command-line interface (CLI) and an API.

One of the main reasons I chose Ollama was its ease of use. When I previously tried to set up some LLMs manually, I had to grapple with the complexity of CUDA libraries, dependency management, and Python environments. Ollama abstracts this infrastructure layer, allowing me to focus directly on the model itself. From my experience with container orchestration using Docker Compose, I know that the simpler the infrastructure, the easier it is to focus on the main task. Ollama seems to embrace this philosophy exactly.

Another significant advantage is its extensive model library. Being able to easily access different versions of popular models like Llama 2, Mistral, and Gemma was a big plus for me. This provides the flexibility to quickly test and switch between the most suitable models for different tasks. For example, in one of my projects, I wanted to quickly see how different models responded to prompt engineering. The variety offered by Ollama significantly accelerated my trial-and-error process. Cross-platform support is also a bonus; being able to use the same tools on my Linux servers as well as my own development environment ensures consistency.

Step 1 & 2: Ollama Installation and First Model Download

The first step to running a local LLM is to install Ollama on our system. While this process varies depending on your operating system, it’s quite simple with the single-line commands provided by Ollama. Since I usually work on Linux servers, I’ll show the installation steps for Linux here. After installation, our first task will be to download an LLM model.

Step 1: Install Ollama

Installing Ollama for Linux systems is quite easy. Open your terminal and run the following command:

curl -fsSL https://ollama.com/install.sh | sh

This command downloads and runs Ollama’s installation script. The script checks for necessary dependencies for your system and installs the Ollama binary. Once the installation is complete, you can start using the ollama command in your terminal. When I performed this installation on a VPS, I realized that if you don’t have a GPU or if it’s weak, you need to choose models that will run on the CPU and adjust your performance expectations accordingly. This is a critical detail, especially on servers with limited resources.

After installation, you can use the ollama --version command to verify that Ollama is running correctly:

ollama --version

If you see a version number as output, Ollama has been successfully installed. After completing this step, you are ready to take your first step into the world of LLMs. For me, such simple installations save time for trying out new technologies and quickly building prototypes.

Step 2: Download Your First LLM Model

Once Ollama is installed, it’s time to download your first LLM model. Ollama hosts many variants of popular models in its library. The ollama run command automatically downloads and runs the specified model. For example, to download the Llama 2 model and start using it immediately, you can use the following command:

ollama run llama2

When you run this command, Ollama first checks if the llama2 model is on your system. If not, it downloads it from the internet. Since the model size can be several gigabytes, this process may take some time depending on your internet connection speed. Once the download is complete, the model will immediately start running in an interactive chat session.

During the model download process, you will see a progress percentage in the terminal. Once the download is complete, you will see a message indicating that the model has been loaded, followed by a prompt (>>>). You are now ready to chat with your local LLM. This is your first moment running an LLM on your own computer, without relying on cloud APIs.

Step 3 & 4: Chatting with Your Local LLM and Model Management

After completing the Ollama installation and downloading our first model, we can now interact with this model and learn how to manage different models. When developing an AI-powered feature for the backend of my side product, testing how different models responded to the same prompt and quickly switching between them was very valuable to me.

Step 3: Chat with Your Local LLM

The ollama run command allows you to start an interactive chat session with a model you’ve downloaded. Let’s say you’ve downloaded the llama2 model. Run the following command in your terminal:

ollama run llama2

Once the model is loaded, you will see the >>> prompt. You can interact with the model by typing your questions or commands here. For example:

>>> Merhaba, nasılsın?

The model’s answer will appear immediately below. In this session, you can get information from the model, ask it to generate text, or use it for different tasks like writing code. To exit, simply press Ctrl+D. This simple interaction is a great starting point for understanding the power and flexibility of a local LLM. I frequently tested these interactions for a feature in my self-developed task management application that allows users to create tasks in natural language.

The model’s response times will vary depending on your system’s hardware and the size of the model you’ve chosen. Smaller, 7B parameter models generally respond faster, while larger models like 13B or 70B require more computational power and may have longer response times. This difference can be noticeably felt, especially on older hardware.

Step 4: Manage Your LLM Models

Ollama provides commands to easily manage your downloaded models. To see which models are on your system, you can use the ollama list command:

ollama list

This command will list all downloaded models, their sizes, and when they were downloaded. If your disk space is limited or you have a model you no longer use, you can remove it from your system. For example, to delete the llama2 model, use the ollama rm command:

ollama rm llama2

This command removes the specified model from your system and frees up disk space. When working with multiple models, these management commands become quite practical. While doing AI-powered inventory optimization in a manufacturing ERP, I frequently used these listing and removal operations to test different models and select the most suitable one. This way, I prevented unnecessary models from taking up disk space and used resources efficiently.

# List all models
ollama list

# Delete a specific model
ollama rm mistral

# Download a new model (if not already listed)
ollama pull codellama

These steps allow you to effectively manage models in your Ollama environment. You can download and experiment with different models according to your needs, and remove them when you’re done.

Step 5: Advanced Usage Scenarios and API Integration

Knowing that Ollama not only offers interaction via CLI but also has a powerful REST API is a huge advantage for developers who want to integrate it into their own applications. Since I frequently use FastAPI in the backend of my side products, integrating with Ollama’s API was a very natural step for me. This way, I can bring the power of local LLMs to my web applications or other systems.

Application Development with Ollama API

Ollama provides a REST API by default at http://localhost:11434. Through this API, you can perform operations such as loading models, text completion, and creating embeddings. For example, we can use the curl command to send a text completion request:

curl http://localhost:11434/api/generate -d '{
  "model": "llama2",
  "prompt": "Bugünkü hava durumu nasıl?"
}'

This command sends the question “How is the weather today?” to the llama2 model and receives the answer in JSON format. This flexibility allows you to develop your own applications with Python, Node.js, or any other programming language. In my Android spam blocker application, I considered this type of API integration to analyze incoming SMS messages with a local LLM and flag potential spam. Since the data would not leave the phone, privacy would remain at the highest level.

To give a simple example with Python:

import requests
import json

def generate_text_ollama(prompt, model="llama2"):
    url = "http://localhost:11434/api/generate"
    headers = {"Content-Type": "application/json"}
    data = {
        "model": model,
        "prompt": prompt,
        "stream": False # Set to True to receive streaming responses
    }
    try:
        response = requests.post(url, headers=headers, data=json.dumps(data))
        response.raise_for_status() # Catch HTTP errors
        result = response.json()
        return result.get("response")
    except requests.exceptions.RequestException as e:
        print(f"API request failed: {e}")
        return None

# Usage example
if __name__ == "__main__":
    question = "How do you start a service in Linux?"
    answer = generate_text_ollama(question, model="llama2")
    if answer:
        print(f"Question: {question}")
        print(f"Answer: {answer}")

This Python script demonstrates how easy it is to send a request to the Ollama API and get a response from the model. In a manufacturing ERP, I can imagine such integrations for automations like querying the status of a specific order in natural language or generating instructions for a production step.

Enhancing with Retrieval-Augmented Generation (RAG)

Combining local LLMs with Retrieval-Augmented Generation (RAG) architectures can significantly improve response quality by providing the model access to up-to-date and specific information. When generating content for my technical blog or analyzing data on my anonymous Turkish data platform, I saw that the LLM needed to interact not only with general knowledge but also with data from specific documents.

RAG essentially involves three steps:

1. Retrieval: Documents containing information relevant to the user query (e.g., internal company documentation, database records) are retrieved from a vector database.
2. Context Augmentation: The retrieved documents are added as context to the prompt sent to the LLM.
3. Generation: The LLM generates a response using the enriched prompt.

Ollama facilitates the “Retrieval” part of this RAG flow by offering embedding models. By generating embeddings locally and storing them in a vector database, it’s possible to set up a completely offline RAG system. This approach is ideal for security and privacy, especially for systems working with sensitive corporate data.

graph TD;
    A["User Query"] --> B["Generate Embedding (Ollama API)"];
    B --> C["Vector Database (Pinecone/Chroma/Weaviate)"];
    C --> D{"Relevant Documents?"};
    D -- Yes --> E["Prompt Enrichment"];
    D -- No --> F["Send Directly to LLM"];
    E --> G["Local LLM (Ollama)"];
    F --> G;
    G --> H["Response to User"];

This architecture allows the LLM to reason not only with the information it has learned but also with the current and specific data we provide. For regulatory queries on a bank’s internal platform, such a RAG architecture can ensure access to accurate and up-to-date information.

Challenges and Solutions I Encountered in Local LLM Setup

Setting up a local LLM isn’t always a smooth experience. My experiences with my own systems and client projects have shown that there are some common challenges and ways to overcome them. Similar situations to the performance and resource management issues I encountered when configuring PostgreSQL settings or choosing Redis OOM eviction policies also apply in the LLM world.

Resource Constraints and OOM Errors

One of the most common problems is insufficient system resources. LLMs, especially large models, consume significant amounts of RAM and GPU memory. Initially, when I tried to run a 13B parameter model on a server with 8GB RAM, I saw that the system was quickly OOM-killed. When I examined the journald logs, there were continuous memory shortage messages. This situation reminded me of resource management problems I encountered when setting cgroup limits or blacklisting kernel modules (e.g., disabling modules like algif_aead).

Solution:

• Model Selection: Preferring smaller parameter (7B or 3B) and quantized models significantly reduces memory consumption.
• System Upgrade: Increasing RAM and GPU (VRAM) capacity, if possible, is the most effective solution.
• Swap Space: Setting up sufficient swap space on Linux systems can prevent the system from crashing in emergencies, but it degrades performance. This is more of a temporary fix.
• cgroup Limits: By assigning memory limits to the Ollama service via systemd units, we can prevent other services from being affected and provide a more stable operating environment. For example, setting soft or hard limits with MemoryHigh or MemoryMax settings.

Performance Bottlenecks

Very long model response times or constant stuttering are signs of performance bottlenecks. This usually occurs when performing CPU-based inference or when the GPU is insufficient. When designing real-time dashboards in a manufacturing ERP, I frequently encountered performance obstacles such as slow database queries or N+1 query problems. LLMs also have a similar need for optimization.

Solution:

• GPU Usage: If available, ensure Ollama can use the GPU. It usually detects automatically, but up-to-date drivers are important.
• Model Quantization: Quantized models with lower bit depth (e.g., 4-bit) can run faster with fewer resources, but sometimes a slight decrease in accuracy may occur. This is a trade-off.
• Connection Pool Tuning: If many applications are using the Ollama API, correctly tuning client-side connection pools reduces the overhead of establishing a new connection for each request. Just like PostgreSQL connection pool tuning.
• Nginx Reverse Proxy: Placing an Nginx reverse proxy in front of the Ollama API can improve performance by providing HTTP/2 support, SSL offloading, and even simple load balancing (if there are multiple Ollama instances).

Model Selection and Accuracy Balance

Different LLM models are optimized for different tasks, and their performance varies. When choosing a model, it’s important to consider not only its size but also its training data and intended use case. For example, Codellama may yield better results for code generation, while Mistral or Llama 2 may be more suitable for general chat.

Solution:

• Use Case-Oriented Selection: Choose a model based on the task you will perform. You can quickly try different models with ollama run <model_name>.
• Community Feedback: Review community feedback and benchmark results for models in Ollama’s model library.
• Experimental Approach: Test different models with a small dataset or prompt group to find the most suitable one for your project. Sometimes a smaller model can perform better for a specific task than a larger, resource-intensive model.

These challenges and solutions demonstrate the importance of not only setting up local LLMs but also running them efficiently and reliably. As a system architect, paying attention to such details is critical for the success of my projects.

Conclusion and Next Steps

Setting up your own local Large Language Model (LLM) with Ollama is much easier and more accessible than you might think. The five-step process we covered in this post offers a strong starting point for anyone who wants to maximize data privacy, avoid cloud API costs, and gain full control over models. As I’ve seen from my own project experiences, local LLMs can be an indispensable solution, especially for applications that handle sensitive data or require offline capabilities.

Ollama’s ease of use, extensive model library, and API integration capabilities make it an attractive option for both individual developers and enterprise environments. Now you can develop intelligent applications on your own servers or workstations, without external dependencies. Remember that, like any technological choice, local LLMs have their own challenges; however, with proper planning and resource management, these obstacles can be overcome.

For your next steps, I recommend continuing to experiment with different models using Ollama, developing your own prompt engineering techniques, and even exploring more advanced architectures like Retrieval-Augmented Generation (RAG). Fine-tuning models with your own datasets or creating your own model for a specific use case will fully unlock the potential of this technology. Local LLMs are an important step towards digital independence, and Ollama provides you with a powerful tool on this journey.