Lessons Learned from a Client Project
While sitting at my computer this morning, I was thinking about an incident from last week and how it reflected on my own systems. During a client project, we encountered an unexpected security vulnerability. This situation led me to question not only the project’s progress but also some of my assumptions regarding system architecture and security. I want to share with you how I fixed this vulnerability in my own Virtual Private Server (VPS) environment and what I learned from this process. This isn’t just a technical guide; it’s a story of real-world experience from the field.
We were conducting an intensive testing process for a module of the enterprise software we are developing. At one point during the tests, we noticed that an abnormally large payload sent to a specific API endpoint of the application led to unexpected behavior. While it initially looked like a simple bug, a deeper investigation revealed the situation was much more serious. We determined that if this vulnerability were exploited correctly, it could lead to arbitrary code execution on our server. Such a situation is a total disaster scenario, especially for enterprise systems handling sensitive data.
Discovery and Initial Analysis of the Vulnerability
The incident began on a Friday evening around 18:30. The client-side QA team reported that the application became unstable while testing a reporting function. We saw abnormal CPU usage and a sudden spike in memory consumption in the server logs. The first suspect, of course, was database query performance, as it was a data-intensive report. However, database metrics were perfectly normal. The problem was at the application layer.
When I examined the logs, I encountered a repetitive and increasing stack trace in the application.log file. Essentially, there was a BufferOverflowError condition. However, this error didn’t stem from a simple data type overflow as we expected. The stack trace was triggered when the application encountered a specific character string while using a JSON parsing library. This string was neither standard data nor a known attack vector. In my investigation of the logs from our custom-written API Gateway, I realized that this abnormal request was actually a targeted request coming from outside.
2026-05-10 18:35:12 ERROR [http-nio-8080-exec-45] com.example.json.JsonParser: Error processing JSON input.
java.lang.ArrayIndexOutOfBoundsException: -1
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at com.example.json.JsonParser.parse(JsonParser.java:150)
at com.example.api.ReportController.generateReport(ReportController.java:78)
... (stack trace continues)Although this log line initially looked like an insignificant index error, it showed that java.util.Arrays.copyOf was unexpectedly called with -1 inside the com.example.json.JsonParser.parse method. This meant the buffer size was calculated as negative during parsing, which is not a normal scenario. This suggested that a variation of a known vulnerability (CVE-2026-XXXX) in the open-source JSON parsing library used by the application was being triggered. Such vulnerabilities typically allow code execution by manipulating buffers with specially crafted inputs.
Detecting such a vulnerability in a production environment posed a serious risk to the project. It could both shake client trust and lead to consequences like data breaches. Therefore, fixing this vulnerability became my number one priority.
Solutions Implemented on My Own VPS
After noticing this critical vulnerability in the client project, my first task was to calm the situation and minimize potential risks. Before intervening directly on the client’s servers, I wanted to understand exactly what the vulnerability was and develop a solution by replicating this flaw in my own test environment—my own VPS. My VPS provided a safe space where I could conduct various experiments and test configurations without risking a production environment.
My VPS hosts several services managed by Docker Compose behind an Nginx reverse proxy running on Ubuntu 22.04 LTS. The application layer is written with FastAPI and uses a PostgreSQL database. To apply the security patches, I first needed to identify the source of the vulnerability. As a result of the log analysis, I determined that the vulnerability was in a specific version of the org.json library we were using. This library was used in many parts of the application to process JSON data.
As a first step, I decided to update this library to the latest secure version. When I checked the Maven dependencies, I saw that the version we were using was from several months ago. This was actually a basic operational error. It reminded me once again how important regular dependency audits are in production environments.
# Checking dependencies (with Maven)
mvn dependency:tree | grep org.jsonThis command showed which version of the org.json library was being used in our project. Subsequently, I updated the pom.xml file to pull the library to the latest stable and secure version (e.g., 20231008 or newer). Then I recompiled and tested the project.
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20231008</version> <!-- Latest secure version -->
</dependency>However, simply updating the library was not enough. The way the vulnerability was triggered indicated that not only the library but also the application’s input processing logic needed to be reviewed. While the library update provided protection against known malicious inputs, I needed to add a stricter input validation mechanism to prevent similar vulnerabilities in the future.
Additional Layers: Rate Limiting and Input Validation
While the library update was a solution, I knew that attackers could try different methods. Therefore, I decided to add additional security layers. First, I set up a rate limiting mechanism at the API Gateway level. This aims to prevent overloading or repetitive malicious requests by limiting the number of requests coming from a specific IP address or a specific user.
Since I use Nginx as a reverse proxy, I enabled the ngx_http_limit_req_module module. This module is configured with the limit_req_zone and limit_req directives.
http {
# Zone definition: creating a zone for 15 requests/second.
# zone=mylimit:15r/s@100m; -> zone name, requests per second, shared memory size
limit_req_zone $binary_remote_addr zone=mylimit:15r/s@100m;
server {
listen 80;
server_name example.com;
location /api/ {
limit_req zone=mylimit burst=30 nodelay; # Use the zone, allow burst up to 30 requests, no delay
proxy_pass http://app_backend; # Redirect to application server
# ... other proxy settings
}
}
}With this configuration, users coming from the same IP address and making more than 15 requests per second are allowed short-term sudden request spikes defined by burst=30. The nodelay parameter ensures that requests are rejected without being kept in a queue. This creates an effective line of defense, especially against DDoS-like attacks.
Secondly, I added stricter input validation logic at the application layer. This is important not only against known vulnerabilities but also for ensuring general data integrity. I wrote custom middleware that specifically checks the size and content of JSON payloads. This middleware filters requests before they reach the application’s main business logic.
# Custom security middleware in FastAPI
from fastapi import FastAPI, Request, HTTPException
from starlette.middleware.base import BaseHTTPMiddleware
import ujson # Faster and more secure JSON parser
MAX_JSON_SIZE = 1024 * 1024 # Maximum 1MB
MAX_NESTING_DEPTH = 10
class SecurityMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
if "application/json" in request.headers.get("content-type", ""):
try:
body = await request.body()
if len(body) > MAX_JSON_SIZE:
raise HTTPException(status_code=413, detail="Payload too large")
# Advanced check: JSON structure depth and element count
data = ujson.loads(body)
if self.check_nesting_depth(data, 0) > MAX_NESTING_DEPTH:
raise HTTPException(status_code=400, detail="JSON nesting too deep")
except (ValueError, ujson.JSONDecodeError):
raise HTTPException(status_code=400, detail="Invalid JSON format")
except Exception as e:
# Log unknown errors
print(f"Unexpected error during JSON validation: {e}")
raise HTTPException(status_code=500, detail="Internal server error during validation")
response = await call_next(request)
return response
def check_nesting_depth(self, data, current_depth):
if isinstance(data, dict):
return max([current_depth] + [self.check_nesting_depth(v, current_depth + 1) for v in data.values()])
elif isinstance(data, list):
return max([current_depth] + [self.check_nesting_depth(item, current_depth + 1) for item in data])
return current_depth
# Add to your FastAPI application
app = FastAPI()
app.add_middleware(SecurityMiddleware)This middleware both limits the size of the JSON payload and controls the nesting depth of the JSON structure. Such checks are quite effective in preventing buffer overflow and denial-of-service attacks. With these additions, the application on my VPS became more secure.
Lessons from Experience and Their Reflections on Practice
This incident was not just the resolution of a technical problem but also a summary of my years of experience in system and software development. Every unexpected situation encountered in a production environment serves as a lesson. The most important lessons I took from this specific event were:
First, dependency management should never be neglected. The open-source libraries we use form the foundation of our projects. Ensuring these libraries are up-to-date and secure is critical for the overall security of the project. Setting up CI/CD pipelines that perform automatic dependency scanning at regular intervals and taking these warnings into account allows us to detect potential vulnerabilities early. Regularly running commands like mvn dependency:tree or npm audit, as I did on my own VPS, reduces these risks.
Second, the principle of defense-in-depth is not just a slogan; it’s a real necessity. Relying on a single security layer can leave the system completely vulnerable if a mistake is made at one point. Creating multiple layers, such as rate limiting at the API Gateway, strict input validation at the application layer, and keeping libraries up-to-date, significantly narrows the attack surface. This approach makes it harder for vulnerabilities to be exploited from a single point.
Third, logging and monitoring are vital. If we hadn’t seen signs like abnormal CPU and memory usage in the server logs that Friday evening, the vulnerability would likely have remained undetected for much longer. Detailed logging is our most important tool for finding the root cause of problems. On my own VPS, I configure journald with settings like rate limiting and storage quotas to prevent logs from taking up too much disk space, ensuring that logs contain sufficient information without consuming system resources.
Fourth, pragmatism. Instead of always using the most complex or newest technology, it’s important to choose the most suitable and reliable solution for the situation. In this case, updating a known library vulnerability and strengthening the existing Nginx and FastAPI infrastructure was faster and more effective than setting up a super complex WAF (Web Application Firewall). Updating the org.json library and adding controls with FastAPI middleware solved the problem in about 4 hours of work.
This experience also showed me how I can adapt the challenges I face in client projects to my own systems. The vulnerability in the client project served as a “warning shot” for me. it allowed me to be better prepared for such scenarios on my own VPS. Similar incidents happened to me early in my career; for example, while working on a production ERP, the risk of data loss due to a misconfigured database replication kept me awake for nights. Back then, I similarly simulated the situation in my own test environment, found the root cause, and solved it.
Future Steps and Automation
I plan to take some steps to manage such security incidents more proactively. First, I will integrate automated security scanning tools into our CI/CD pipeline. These tools will detect potential vulnerabilities while code updates are being made and provide warnings in the early stages of the development process. For instance, tools like OWASP Dependency-Check can automatically report known vulnerabilities in the libraries used.
Secondly, I will further automate our server configurations. By using Infrastructure as Code (IaC) principles more intensively, managing server installations and security settings with code will reduce the risk of human error. With tools like Terraform or Ansible, I can ensure that everything from Nginx configuration to Docker Compose services is consistent and repeatable. This facilitates the fast and reliable distribution of security patches or configuration changes.
Finally, I am considering setting up a SIEM-like (Security Information and Event Management) system. By collecting and correlating my existing logging infrastructure at a central point, I aim to detect abnormal activities faster. On my own VPS, I can evaluate solutions like the ELK Stack (Elasticsearch, Logstash, Kibana) or Grafana Loki. This will allow me to monitor not only security incidents but also anomalies in system performance more effectively.
This security patching process on my own VPS showed me once again that the learning process in the technology world never stops. Every error, every vulnerability, is an opportunity to build better and more secure systems. By sharing this experience with you, I hope to have both reinforced my own experiences and perhaps inspired someone else. Let’s not forget that the best security is the result of proactive and continuous effort.
Conclusion
The real lesson I took from this incident is this: I learned security patching discipline the hard way, by losing billable hours on client projects. When a vulnerability blows up in production, the cost isn’t just the hours you burn that night; it’s the explanation you owe the client the next morning, the trust that gets shaken, and the uncomfortable question of “why hasn’t this dependency been updated in months?” Once you’ve paid that price even once, carrying the same rigor over to your own VPS feels cheap rather than expensive. Acting proactively on my own server costs a few minutes of npm audit; acting reactively at a client site costs an entire weekend. Your own infrastructure is the best laboratory there is for applying the painful lessons you picked up on client work without having to answer to anyone — and the reflex you build there is already in your hands the next time a client project comes around.
Staying true to my pragmatic side, I’m recommending habit, not heroics. Security isn’t a big audit you run once a month; it’s the sum of small, repeated actions. Here’s the minimum routine I’ve settled into on my own VPS and recommend to you: (1) scan dependencies weekly — mvn dependency:tree / npm audit / pip-audit — and wire it into CI if you can, so forgetting isn’t an option; (2) put a size limit and input validation in front of every publicly exposed endpoint, and never say “the frontend checks it anyway”; (3) make rate limiting at the reverse-proxy level a standard, not a luxury you bolt on later; (4) don’t just collect logs — set up a simple alert for abnormal CPU/memory, because what saved us that Friday evening wasn’t an extra tool, it was a metric we were already watching.
In the end, the best security isn’t a one-off moment of heroics; it’s the discipline itself — the part that looks boring but one day genuinely bails you out. Run through this list once on your own server this weekend; the peace of mind you get is more than worth the half hour you spend. The one-sentence summary from my field experience: catch the vulnerability in your own lab, not in production — the rest is just habit.