Go Paperless: Make All Your Documents OCR-Searchable

Recently, I turned my house upside down, rummaging through all the folders to find an electricity bill from 2023. After about 45 minutes of searching, I realized the bill was actually in my email inbox – but it wasn’t searchable either. This kind of experience was one of the main reasons that pushed me to switch to a document management system like Paperless-ngx. Paperless-ngx is an open-source and highly effective solution that uses OCR (Optical Character Recognition) to convert all your physical and digital documents into text-based, easily searchable digital archives.

Thanks to this system, I can find an old warranty card, a bank statement, or any receipt in seconds. It eliminates the risk of losing physical documents while making digital clutter manageable. Paperless-ngx significantly boosts productivity, especially for home office users, small businesses, or anyone like me who wants to organize their personal documents.

What is Paperless-ngx and Why is it Critical for a Digital Archive?

Paperless-ngx is essentially a document management system (DMS). It allows us to ingest physical documents by scanning them or by directly uploading digital ones. However, its most important feature that distinguishes it from other solutions is that it automatically performs OCR on all these documents and indexes the extracted text, making it searchable. This means you can find a company name, amount, or date on an old scanned invoice by simply typing it into the search bar.

For me, Paperless-ngx is more than just a “paperless office” tool; it means transforming years of accumulated information into a meaningful data pool. Instead of wondering where a document is, directly searching for what I need saves significant time in my daily workflow. Furthermore, considering the restrictions GDPR and similar regulations place on personal data management, being able to quickly access and, if necessary, delete specific documents is a major advantage.

Installation Process: Quick Start with Docker Compose

The easiest way to install Paperless-ngx, and my preferred method, is using Docker Compose. This method offers great flexibility in managing dependencies and running the system’s different components (web server, database, broker, OCR engine) in an isolated manner. I usually use this method on my own VPS or a local server.

First, you need to have Docker and Docker Compose installed on your server. If not, you can quickly install them with the following commands. Since I usually prefer Ubuntu Server LTS versions, the commands will be tailored accordingly.

# Docker installation
sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io -y

# Docker Compose installation (often easier to manage with Python pip)
sudo apt install python3-pip -y
pip install docker-compose

After installing Docker and Docker Compose, you need to create a directory for Paperless-ngx and place the docker-compose.yml file inside it. I usually use a directory like /opt/paperless.

mkdir /opt/paperless
cd /opt/paperless
wget https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/docker-compose.yml
wget https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/docker-compose.env

Open the docker-compose.env file and adjust variables like PAPERLESS_OCR_LANGUAGE according to your needs. For example, for Turkish documents, setting PAPERLESS_OCR_LANGUAGE=tur+eng would be logical. Also, don’t forget to configure important security settings like PAPERLESS_URL and PAPERLESS_SECRET_KEY. Make database passwords like POSTGRES_PASSWORD strong and unique.

Finally, to bring the system up:

docker compose pull
docker compose up -d

These commands will download the necessary Docker images and start the Paperless-ngx services in the background. The system is expected to be ready within approximately 5-10 minutes (depending on your internet speed and server performance). Initial setup involves some database and indexing operations.

How Do We Upload Documents to Paperless-ngx?

There are multiple ways to upload documents to Paperless-ngx. This variety is something I really appreciate, as it adapts to different usage scenarios. Whether manual or automatic, you can easily integrate all kinds of documents into the system.

1. Manual Upload via Web Interface: This is the simplest method. After logging into the Paperless-ngx interface through your browser, you can drag and drop files using the “Upload” button in the “Documents” section. This is practical, especially when uploading a small number of documents or performing a one-time operation. I usually use this method when I need to instantly add an urgent invoice or a signed contract to the system.

2. Automatic Upload with the “Consume” Folder: This is one of Paperless-ngx’s most powerful features. The system continuously monitors a specific folder (usually /usr/src/paperless/consume or another folder you’ve bind-mounted as a Docker volume). It automatically detects, OCRs, and archives every document you place in this folder. I use a script that saves all documents I scan with my scanner directly to this folder. For example, I use an hp-scan command to put PDFs from the scanner directly into this folder.

   # Example of saving a scanned document from your scanner to the 'consume' folder
   # This command may vary depending on your scanner model and software
   scanimage --format=pdf > /opt/paperless/consume/new_document_$(date +%Y%m%d%H%M%S).pdf

   Thanks to this automation, digitizing a physical document becomes as easy as pressing a button on the scanner.

3. Email Integration: Paperless-ngx can also automatically retrieve and process attachments sent to a specific email address. This feature offers an excellent solution, especially for invoices, subscription notifications, or contracts received via email. For example, you can fully automate the process by setting up a mechanism to filter a Gmail account to a specific folder and then direct the attachments of emails arriving in that folder to Paperless-ngx’s consume folder. I can do this with fetchmail or a similar tool.

   # Example of pulling attachments from an email inbox with fetchmail (can be complex)
   # This is usually run as a cron job
   # Your fetchmailrc file should have POP3/IMAP settings and direct emails to a script
   # For example: program "/usr/local/bin/email_to_paperless.sh"

   This method is similar to an approach I use for automatically processing customer invoices in one of my side products. I saw once again how important it is to digitize document flow here.

OCR and Automatic Tagging: Magic Under the Hood

At the heart of Paperless-ngx lies the Tesseract OCR engine and advanced automatic tagging algorithms. When a document is uploaded to the system or dropped into the consume folder, Paperless-ngx immediately springs into action.

OCR Process:

1. Text Extraction: The document is sent to the Tesseract OCR engine. The engine recognizes and extracts text from the document. Thanks to its Turkish character support, it yields quite successful results for Turkish users like me.
2. PDF/Image Integration: If you uploaded a PDF, the text obtained via OCR is added as a hidden layer within the original PDF. This allows you to search for text even in your PDF reader. For image documents, the text is stored in the database.
3. Indexing: All extracted text is indexed for full-text search in the database. This enables you to search through thousands of documents in seconds.

Automatic Tagging: The text from OCR is analyzed by Paperless-ngx’s “Matching Algorithms.” These algorithms allow documents containing specific keywords or text patterns to be automatically assigned a “Correspondent,” “Document Type,” and “Tags.”

For example, you can automatically assign “Turkcell” as the correspondent to every document containing the word “Turkcell.” Or, you can assign “Invoice” as the document type to every document containing the word “invoice.” In my experience, setting up this automation takes some time initially, but in the long run, it incredibly reduces the document management burden. Approximately 85% of my 1500 documents are automatically tagged thanks to these rules.

# Example of defining an automation rule in the Paperless-ngx interface (pseudo-code)
# Rule: If content contains "Turkcell", then assign correspondent "Turkcell"
# Rule: If content contains "elektrik faturası", then assign tag "Elektrik"
# Rule: If content contains "garanti belgesi" and "tarih" (regex), then assign document type "Garanti"

The more detailed you define these rules, the greater the system’s automation capability. You can also create much more complex matching rules using regular expressions (regex). The automation engine I used on an internal platform for a bank also worked with similar logic; it routed documents to relevant departments based on their content.

Archive Management and Search Features: Lost Documents Are a Thing of the Past

With Paperless-ngx, you don’t just digitize your documents; you also intelligently organize them and find them instantly when needed. For me, the biggest benefit is no longer having to wonder “where was that document?”

Advanced Search: Paperless-ngx’s search feature is quite powerful. In addition to simple keyword searches, you can search within specific fields (correspondent, document type, tag) or specify date ranges. For example:

• invoice Turkcell (Documents containing both “invoice” and “Turkcell”)
• type:invoice correspondent:vodafone (Invoices from Vodafone)
• tag:warranty after:2024-01-01 (Documents tagged “warranty” after 2024-01-01)
• content:contract AND content:signature (Documents containing “contract” and “signature” in their content)

This flexibility reduces the time spent finding a specific document to almost zero. Need an old tax return? Just type “tax return 2023” into the search bar. Considering I used to pull reports with complex queries in an ERP project, this simplicity surprised me.

Saved Views: You can save frequently used search queries as “Saved Views.” For example, you can create views like “Last 30 Days’ Invoices” or “Pending Process Documents” to access relevant documents with a single click. This acts like a custom reporting mechanism and significantly speeds up my workflow. I typically use saved views to track monthly expense invoices or documents related to a specific project.

Document Linking: Paperless-ngx also allows you to link documents to each other. For example, you can link a product’s invoice to its warranty certificate or different appendices of a contract, making it easy to jump to related documents. This is very useful, especially in complex projects or situations with multiple interrelated documents.

Tips for Secure and Efficient Use of Paperless-ngx in a Production Environment

While Paperless-ngx is a great tool for personal use, if you’re going to store more critical data or plan to use it with a small team, you absolutely must take some security and efficiency precautions. Here are a few critical points I’ve learned from my 20 years of system administration experience.

1. Backup Strategies: The most important issue. Your data is more valuable than the system itself. Paperless-ngx stores its data in Docker volumes. You need to regularly back up these volumes. A basic backup command I use looks something like this:

   # Stop Paperless-ngx containers (for database consistency)
   docker compose down
   
   # Backup data volumes
   tar -czvf /mnt/backups/paperless_data_$(date +%Y%m%d).tar.gz /opt/paperless/data /opt/paperless/media
   
   # Restart Paperless-ngx containers
   docker compose up -d

   You should automate these backups daily or weekly with a cron job and move the backups to a different storage location (e.g., S3-compatible storage or another server). When I experienced a disk failure last year, regular backups allowed me to recover with only 2 hours of data loss.

2. Nginx Reverse Proxy and SSL: Paperless-ngx runs over HTTP by default. To secure access to the web interface, you should set up an Nginx reverse proxy and use an SSL/TLS certificate with Let’s Encrypt. This is critically important, especially if you’re uploading sensitive documents to the system.

   # Nginx config example (simplified)
   server {
       listen 80;
       server_name your_paperless_domain.com;
       return 301 https://$host$request_uri;
   }
   
   server {
       listen 443 ssl http2;
       server_name your_paperless_domain.com;
   
       ssl_certificate /etc/letsencrypt/live/your_paperless_domain.com/fullchain.pem;
       ssl_certificate_key /etc/letsencrypt/live/your_paperless_domain.com/privkey.pem;
   
       location / {
           proxy_pass http://localhost:8000; # The port Paperless-ngx is running on
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }

   This configuration encrypts all traffic and provides protection against potential man-in-the-middle attacks.

3. Resource Limits and Optimization: Defining CPU and memory limits for containers in the Docker Compose file prevents other services on your server from being affected. This is especially important as the OCR container can consume high resources.

   # Example of resource limits in docker-compose.yml
   services:
     webserver:
       # ...
       deploy:
         resources:
           limits:
             memory: 2G
           reservations:
             memory: 1G
     tesseract:
       # ...
       deploy:
         resources:
           limits:
             cpus: '2.0'
             memory: 4G
           reservations:
             cpus: '1.0'
             memory: 2G

   These limits prevent Paperless-ngx from competing with other applications on your server or suddenly consuming all resources. I also use similar cgroup limits on Docker Swarm for the backend of my side products.

4. User and Permission Management: If multiple people will be using Paperless-ngx, you should enforce strong password policies and configure permissions to ensure users only have access to the documents they need. Paperless-ngx offers user groups and document-based permission management.

Challenges Encountered and Solutions: My Experience

Although Paperless-ngx is a fantastic tool, I also encountered some challenges during setup and use. These challenges usually stemmed from “edge case” situations inherent in open-source projects or misconfigurations.

1. OCR Performance on Large PDF Files: It’s normal for the OCR engine to use 100% CPU for significant periods when processing PDF files 200 MB or larger. Once, when I uploaded a 1000-page scanned book PDF, the OCR process took over 15 minutes.

   • Solution: For such large files, it’s best to split them into smaller parts or accept that the OCR process runs in the background and be patient. Additionally, allocating more CPU resources to your server or more cores to the tesseract container (like the deploy.resources.limits.cpus example above) can improve performance.

2. Incorrectly Recognized Text (OCR Errors): OCR errors are inevitable, especially with low-quality scans or handwritten documents. I once saw some numbers in a bank statement that were incorrectly recognized.

   • Solution: You can manually edit the “Content” section of the document from the Paperless-ngx interface. This will directly affect search results. Using a quality scanner and correctly setting scanning parameters (like DPI) also minimizes errors. I recommend a minimum of 300 DPI, color or grayscale scanning.

3. Disk Space Consumption: Over time, thousands of documents and their OCRed copies can take up significant space on your server. When I archived over 5000 documents, my disk usage exceeded 50 GB.

   • Solution: While performing regular backups, it may be necessary to periodically clean old and unnecessary documents from the archive or switch to a larger disk space. You can monitor disk usage with the du -sh /opt/paperless/data command. Additionally, you can reduce image sizes by enabling settings like PAPERLESS_OPTIMIZE_IMAGES.

4. Database Bloat (PostgreSQL WAL bloat): Paperless-ngx stores documents and OCR text in PostgreSQL. In situations of heavy transaction and deletion, database logs (WAL) can bloat, taking up unnecessary disk space.

   • Solution: Regularly running PostgreSQL’s VACUUM and ANALYZE commands, especially VACUUM FULL, can reduce this bloat. However, VACUUM FULL locks the database, so it should be done during planned maintenance windows. I also experienced a WAL bloat issue in a production ERP, and the solution involved strict autovacuum and checkpoint setting optimization.

Conclusion: The Importance of Document Management in a Digitalizing World

Since I discovered Paperless-ngx, the time I spend on “paperwork” has significantly decreased, and I can find any document I need in seconds. This is not just a time-saver but also provides mental relief. Instead of stressing about where a document is, I can simply focus on my work.

If you, like me, are tired of physical or digital document clutter, I highly recommend trying Paperless-ngx. Its setup is quite easy thanks to Docker, and its management is very efficient due to its automation capabilities. Remember, digitalization is a process that fundamentally changes not only business processes but also our personal productivity. Digitizing your documents is a small but very valuable investment in the future. The next step might be to analyze the data in this digital archive with AI to extract trends; my research on this topic is ongoing.