Technology
Retry Storms: Timeout Budget and Latency Amplification
In distributed systems, badly designed retries make outages worse. An approach to limiting damage with timeout budgets, retry budgets, and backpressure.
#sre
18 posts found.
Technology
Isolating Bad Nodes with Envoy Outlier Detection
Threshold, signal and rollback discipline for Envoy outlier detection — shrinking the blast radius of broken nodes in distributed systems.
Career
Post-Change Verification Cadence: Smoke, SLO, and Rollback
Assuming the release is done is how you summon an incident. A practical framework for turning post-change verification into a cadence: fast smoke checks…
Career
Major Incident Management: Incident Commander and Runbook Practices
In big outages the largest risk isn't technical, it's coordination. How I drive MTTR down with the IC role, a steady comms cadence, and a practical runbook…
Career
Incident Walkthrough and Operational Signals in a Platform Interview
An incident walkthrough framework and scoring rubric for measuring a candidate's real production reflex in SRE/Platform/Infra interviews.
Tutorials
Kubernetes Control Plane Certificate Expiry: A Runbook
When API Server access suddenly breaks with x509 errors; certificate renewal and safe recovery steps for kubeadm-based clusters.
Tutorials
Designing a Telemetry Pipeline with OpenTelemetry Collector
Treating Collector not just as an agent but as a central telemetry backbone for sampling, redaction, routing and multi-destination delivery.
Career
Balancing Operational Confidence and Speed with DORA Metrics
Keeping production confidence while increasing deployment speed: a practical management cadence and team rhythm that combines DORA metrics with SRE signals.
Career
Operational Readiness Review (ORR) Before Go-Live
Turning go-live from 'ship and pray' into something with clear risk, ownership, and rollback reflex: a practical ORR gate and checklist.
Tutorials
SLO-Driven Load Testing with k6: Capacity Baselines and Release Gates
A practical approach that turns load testing from a peak-RPS race into an SLO-driven (latency/error) capacity baseline and a CI release gate.
Career
Managing Operational Debt with a Toil Budget
A toil budget approach for sustainable operations: measuring repetitive manual work, making it visible, and protecting time for improvement.
Technology
Change Brakes via Error Budget: Designing a Release Gate
How do I turn SLO and error-budget signals into a release gate that controls change without halting it? Field-tested thresholds and an operations flow.
Technology
A Safe Experiment Plane for Chaos Engineering
Hypotheses, blast radius and automatic rollback guardrails so resilience tests don't turn into blind risks in production.
Tutorials
Protecting the Kubernetes Control Plane with API Priority and Fairness
A practical APF setup that prioritizes critical traffic and fairly queues noisy callers, lowering the risk of API server overload.
Tutorials
Designing Maintenance Waves for Kubernetes Node OS Patching
Roll out node patches in maintenance waves rather than all-at-once: drain, PDB, parallelism, and a safe rollback path.
Career
Minimum Viable Runbook Template and Incident Decision Points
A minimum template, thresholds, and practical examples for turning the runbook from a documentation pile into a tool that produces decisions during an incident.
Technology
Database Connection Pool Saturation and the Latency Feedback Loop
A practical framework to detect the queue, timeout, and retry loop that emerges when a connection pool clogs, and to intervene safely.
Tutorials
Multi-Point Service Health Monitoring with Blackbox Exporter
An installation guide that pushes a real reachability signal into Prometheus by running HTTP, TCP, and TLS checks from multiple network locations.