When analyzing supply chain data in a production ERP, the idea of sending critical information to a publicly accessible cloud-based LLM always bothered me. Processing sensitive data like proprietary production plans, customer lists, or financial details on third-party servers was an unacceptable risk for me. In such situations, the only way to ensure data security is to run the LLM in a local environment, under our own control.
This is where Ollama comes in. Ollama is a tool that allows you to easily run large language models (LLMs) on your local system. This way, you can set up your own AI assistant without an internet connection and without the risk of leaking your data, securely automating your sensitive business processes. In this post, I will step-by-step explore why Ollama is important, how to install and use it, and the data security advantages it offers.
Why Ollama is Important: Does it Provide Data Security and Control?
The potential of LLMs in modern software development processes and operational workflows is huge, but the privacy and security of corporate data are always a primary concern. Especially in high-security environments like a bank’s internal platform or in the financial calculators of my own side product, I need to process user data without sending it externally. In these scenarios, cloud-based LLM services often create a new risk factor rather than a solution.
Ollama is designed to close this critical data security gap. By running models on your local machine, it prevents your data from leaving your company network or personal computer. This is a vital advantage, especially for those working in sectors subject to data protection regulations like GDPR and KVKK. Furthermore, because it doesn’t require an internet connection, you can continue to benefit from LLM capabilities even in offline environments or during network outages. Since you have full control, you can manage from start to finish which model runs with which data, how much resource the model consumes, and how security policies are implemented. This full control is an indispensable feature for corporate environments.
Ollama Installation: Step-by-Step Guide for Linux Systems
Installing Ollama on my Linux system is generally a quite simple process. For most distributions, a single-line command is sufficient. Since I generally use Ubuntu-based distributions, I will refer to that environment in this guide. Before installation, it’s important to ensure your system meets the minimum requirements; especially sufficient RAM and, if possible, a GPU are critical for models to run performantly.
As a first step, downloading and running Ollama’s official website installation script is the easiest method. This script checks your system, installs necessary dependencies, and automatically sets up the Ollama service.
curl -fsSL https://ollama.com/install.sh | shAfter running this command, the Ollama service will automatically start under systemd. We can use the following command to check the service status:
systemctl status ollamaIf the installation is successful, the output will show a status like active (running). This means Ollama is ready to run in the background. If you encounter any errors during installation, the terminal output usually provides clues such as missing dependencies or permission issues. In this case, you may need to manually install the necessary packages or adjust permissions according to the error message. For example, having GPU drivers correctly installed directly affects performance, especially for large models. If you are using an Nvidia GPU, checking your drivers with the nvidia-smi command is a good habit.
Model Selection and Download: Which LLM is Right for You?
After installing Ollama, the most exciting step is choosing and downloading the LLM model you will use. Ollama allows you to easily download and run many popular open-source models (Llama 2, Mistral, Gemma, Code Llama, etc.). Each model has different sizes, capabilities, and resource requirements. Therefore, choosing the right model can vary depending on your system’s hardware and your intended use.
You can see available models and their abbreviations on Ollama’s official site or with the ollama run --help command. For example, if you’re looking for a general-purpose model, llama2 can be a good starting point. If you’re looking for something more compact and fast, you can try mistral or gemma models. For code generation, codellama would be more suitable.
To download a model, we use the ollama pull command:
ollama pull llama2This command downloads the latest version of the llama2 model. The download process may take some time depending on the model size and your internet connection speed. Models are typically several gigabytes in size, so make sure you have enough disk space. In my experience, I usually start with llama2 or mistral for the initial setup because they offer a good balance for general tasks and don’t consume too many resources. Later, I move on to trying different models for more specific needs.
Local LLM Usage: Prompt Engineering and API Integration
After downloading the model, there are several ways to interact with Ollama. The simplest method is to chat directly via the terminal. You can run any model you’ve downloaded with the ollama run command and immediately start sending prompts.
ollama run llama2
>>> Hello, how are you?This way, you initiate an interactive chat with the model. The model’s response time will vary depending on your system’s hardware and the model’s size. In my experience, even for a simple question, the first response might take a few seconds, but subsequent responses are usually faster.
For more advanced use cases, Ollama provides a REST API. This API allows you to programmatically access your local LLM from your own applications or scripts. For example, when writing a Python application, you can send requests to the Ollama API using the requests library to get responses from the model. This is a method I frequently use in my own side products or in a client project, especially when sensitive data needs to remain within the company.
Let me show you how to send a request to the Ollama API with a simple Python example:
import requests
import json
def generate_text(prompt, model="llama2"):
url = "http://localhost:11434/api/generate"
headers = {"Content-Type": "application/json"}
data = {
"model": model,
"prompt": prompt,
"stream": False # To wait for the full response
}
try:
response = requests.post(url, headers=headers, data=json.dumps(data))
response.raise_for_status() # Catch HTTP errors
result = response.json()
return result["response"]
except requests.exceptions.RequestException as e:
print(f"Error during API request: {e}")
return None
# Example usage
my_prompt = "What is the capital of Turkey?"
response_text = generate_text(my_prompt)
if response_text:
print(f"Model response: {response_text}")This API integration has been a real game-changer for me. Especially in a production ERP, being able to use LLM capabilities without sending data externally for tasks like summarizing free-text notes from operator screens or analyzing error codes has increased operational efficiency while ensuring data security.
Performance Optimization and Resource Management: Speeding Up Local LLMs
When running local LLMs on your own system, performance and resource management become important. Especially large models can consume significant amounts of RAM and, if available, GPU memory. In my system administration experience, I try to manage such resource consumption by adjusting cgroup limits or monitoring journald logs on Linux.
1. Hardware Optimization:
- RAM: LLMs need plenty of RAM as they load model weights into RAM. You can run small models with 8GB RAM, but 16GB or 32GB RAM is preferred for larger models.
- GPU: If you have an Nvidia or AMD GPU, you can achieve much faster response times by running Ollama models on the GPU. GPU memory (VRAM) is also a critical factor here.
- Disk Speed: A fast SSD improves the overall experience when models are loaded from disk.
2. Ollama Configuration: Ollama tries to use the best hardware on the system by default. However, in some cases, especially if you have multiple GPUs or want to use a specific GPU, you can manage this with environment variables.
You can target a specific GPU with a command like export OLLAMA_GPU=0.
3. Linux Resource Management (cgroup):
Let’s say you’re running Ollama as a background service and you don’t want this service to consume the entire system. You can limit the amount of memory a service can use with cgroup directives like MemoryHigh, MemoryMax within systemd units.
In an ollama.service file like the following (usually found at /etc/systemd/system/ollama.service):
[Unit]
Description=Ollama Service
After=network-online.target
[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="OLLAMA_HOST=0.0.0.0:11434"
# Examples for memory limiting
# MemoryHigh=8G # Soft limit, tries to release memory when system is under pressure
# MemoryMax=12G # Hard limit, not allowed to exceed this amount
[Install]
WantedBy=multi-user.targetAfter making these settings, you need to restart the service with systemctl daemon-reload and systemctl restart ollama commands. Such resource management is vital for maintaining overall system stability, especially in hybrid deployment scenarios where other critical services are also running on the same server. Last month, when I accidentally wrote sleep 360 in the backend of one of my side products and caused a service to be OOM-killed, I once again saw how important these cgroup limits are.
Advanced Ollama Usage Scenarios: RAG and Agent Patterns
Ollama is more than just a standalone LLM execution tool; it also forms a solid foundation for more complex AI application architectures. Especially when combined with approaches like Retrieval-Augmented Generation (RAG) and Agent patterns, the capabilities of local LLMs can extend to a much wider area. In my own AI application architecture experiences, I typically use these patterns to increase efficiency and accuracy.
1. Accuracy with Retrieval-Augmented Generation (RAG): RAG is a technique that enables LLMs to generate responses not only by using the general knowledge they have learned but also by incorporating specific data retrieved from an external knowledge base (documents, databases, websites). This is critical for reducing the LLMs’ tendency to “hallucinate” and for producing more accurate responses based on corporate data.
To set up a local RAG system, we can follow these steps:
- Data Source: Collect internal documents (PDFs, Word files, internal wiki pages).
- Generate Embeddings: Convert these documents into numerical vectors using an embedding model supported by Ollama (e.g.,
nomic-embed-text). - Vector Database: Store the generated embeddings in a local vector database (e.g., ChromaDB, Milvus).
- Querying and LLM Integration: When a user asks a question, convert this question into an embedding and find the most relevant document snippets in the vector database. Then, send these snippets and the original question as a prompt to the LLM running in Ollama.
This approach can lead to significant efficiency gains, especially in a production company’s ERP, by digitizing old manual guides and allowing operators to ask questions directly to the LLM. Since the data remains within the company, security concerns are also minimized.
2. Automation with Agent Patterns: Agent patterns position LLMs as “agents” that can use tools (APIs, database queries, code execution environments) to perform specific tasks. For example, an agent can analyze a user’s request and make a call to an ERP API to check stock status or run a Python script to perform a financial calculation.
Creating an agent architecture with a local LLM is ideal for automating sensitive corporate processes. For example, an internal request like “What is the stock status of product X, and if it’s less than 100 units, create an automatic order” can be managed entirely within the company by an agent, a local LLM, an ERP API, and an order management system integration. This is an approach I use to automate complex workflows in my custom financial calculators developed on my own VPS or in side products like an Android spam blocker. Instead of setting up fallback mechanisms between different LLM providers (Gemini Flash, Groq, Cerebras), I can run such systems more controllably and securely with a single local Ollama setup.
Conclusion
Setting up a local LLM with Ollama is a powerful and practical solution for anyone seeking data security and full control. Whether it’s processing sensitive data in a production ERP or protecting privacy in your personal projects, Ollama directly addresses this need. Thanks to its ease of installation and the flexibility it offers, bringing AI capabilities to our own infrastructure is no longer a dream, but a tangible reality.
By following the steps outlined in this guide, you can easily set up your own local LLM. Remember that performance will vary depending on your hardware, so proper management of your system resources is critical. Advanced usage scenarios like RAG and Agent patterns further enhance the potential of your local LLMs, allowing you to get maximum benefit from AI in your corporate or personal projects. In my own experience, the sense of security and control provided by such local solutions outweighs the convenience offered by cloud-based alternatives. The next step should be to consider how to integrate these local LLMs into your existing workflows.