Evidence Collection Kit and Roles During an Incident

There are two extremes during an incident: either there is no data at all, or you drown the system in logs and pcap by trying to “collect everything.” Both end the same way: in the postmortem, the question “what exactly happened?” gets answered with interpretation rather than evidence.

What works best for me in the field: an Evidence Collection Kit + role assignment. The two together don’t just put out the fire; they also make the next incident cheaper.

What are we aiming for?

• Not killing the evidence while resolving the incident
• After the event, making improvements based on evidence, not luck
• Keeping the initial data ready in case a security (forensics) need arises

Role assignment: three people, three responsibilities

Even in small teams, this separation pays off:

1. Incident Commander (IC): makes decisions, sets priorities, manages communication
2. Scribe: keeps the timeline, records the actions taken and their outcomes
3. Evidence Owner: runs the evidence checklist (logs/snapshots/configs)

When all three roles are merged into one person, the evidence side is usually the part that gets lost.

Evidence Collection Kit: “minimal but reproducible”

I split the kit into 6 pieces. For each piece, the answer to “why does this exist?” is clear.

1) Time standard

• NTP/Chrony health check on all systems
• During the incident, all records collected in UTC or one single time standard

Quick check:

timedatectl status
chronyc tracking 2>/dev/null || true

2) Change evidence

The most common root cause: “something just changed.”

Collect:

• Deploy/CI/CD log (commit SHA, pipeline id)
• Config management changes
• Firewall/LB/policy changes

3) Access evidence

Collect:

• Bastion/SSO audit
• Privileged command records (auditd, not shell history; audit)
• Break-glass usage (who, when, why)

4) Service evidence

Collect:

• Error rate, latency, saturation (SLO/SLI)
• The most critical dashboard screenshot or export
• If there is an alarm storm, a note of “which are symptoms, which are causes”

5) Host evidence

Collect:

• Baseline CPU/memory/disk/net metrics
• Kernel logs from the last 30–60 minutes
• Single-line critical signals such as OOM, conntrack table full, filesystem error

Practical:

uptime
free -m
df -h
dmesg -T | tail -n 200
journalctl --since "-60m" --no-pager | tail -n 200

6) Config snapshot (config evidence)

Goal: freeze “the current state.”

• Edge: routing table / policy snapshot
• Application: feature flag state, env/secret version (the version, not the value)
• Infra: LB pool members, weights, health-check state

How does it plug into the incident flow?

The most practical model:

• The IC makes the containment decision in the first 5 minutes
• The Scribe writes every action as “timestamp + command + result”
• The Evidence Owner finishes the kit at the first calm moment after containment

This way, “putting out the fire” and “producing evidence” run at the same time.

Conclusion

The evidence collection kit does not slow down an incident; when set up correctly, it speeds it up. Because in the panic moment, the team does not stop to ask “what should we collect?” — the standard runs on its own. Operational leadership starts here: designing not only the system, but the system that manages the event itself.