In many organisations, operations knowledge nominally sits with the team but in practice rides on the shoulders of a few senior engineers. Runbooks may be written, alarms may be defined, an on-call rotation may even exist; but when a real incident lands, deciding which signal matters, which service should be throttled first and how to ask for help from which team still routes back to the same people. The sustainable answer to this is not just more documentation; it is skill transfer supported by shadow on-call.
Why is shadow on-call necessary?
Because production knowledge is heavily context-bound. Understanding what a log line is really saying, telling apart which alarm is noise, or sensing when to call a rollback is not learned by reading a document. Operations culture forms when you are exposed to the actual flow of events.
Shadow on-call closes this gap:
- The senior engineer runs the production response.
- A less experienced engineer follows the same flow live.
- The reasoning behind decisions is verbalised in the moment.
- A short learning loop is closed after the response.
This model is more than “sitting next to someone and watching”; it is a deliberate teaching design.
What is the right framing for a technical leader?
I prefer to think of shadow on-call in three phases:
- Preparation: system topology, alarm sources and the escalation path are walked through.
- Companion phase: during real events, the shadow engineer follows the decision flow live.
- Handover: in low-risk events, the first response is initiated by the shadow engineer.
Each of these phases needs explicit expectations. Otherwise the process drifts into passive observation and skill transfer stays limited.
Why is writing runbooks alone not enough?
Because a good runbook captures the visible steps, not the invisible reasoning. During an incident, a technical leader is evaluating several things at once:
- Does this alarm have business impact?
- Could this be the first sign of a wider chain of symptoms?
- Should third-party teams be involved now or later?
- Is rolling back, or throttling traffic, the safer move?
This chain of reasoning can be written down, but it only sinks in when it is lived together. That is why shadow on-call sits at the intersection of mentorship and operations.
What is the effect on team processes?
When shadow on-call is set up well, three important results show up:
- The on-call load lifts off a single person and gets distributed more fairly.
- Incident communication becomes more standardised.
- The senior engineer’s internal decision model becomes visible.
At enterprise scale, this is not just a training programme; it is an operational capacity multiplier. As the team grows, the most expensive resource is no longer an expert’s time, it is the ability to replicate that expertise.
Which metrics actually mean something?
Measuring this model only by “how many people have been added to the rotation” is misleading. More useful indicators are:
- The first-correct-response rate on low- and medium-risk incidents
- Average time to escalation
- The quality of new decision notes added to runbooks
- The number of engineers who can draw independent conclusions in post-incident retrospectives
These metrics give a much better view of whether knowledge is actually being transferred.
Why does the technical leader’s language matter?
Skill transfer is not just task transfer; it is trust transfer. If the senior engineer always finishes the response themselves, the team stays dependent on them. If they open up a controlled risk surface instead, the team grows. That is why a technical leader should speak in sentences like these:
- “You build the first hypothesis, I’ll validate.”
- “Walk me through why you classified this alarm as low priority.”
- “You write the response here, and I’ll do the final check.”
This approach preserves mentorship and operational safety at the same time.
Conclusion
In technical leadership, shadow on-call and skill transfer are not just a way of growing junior engineers. The model directly shapes incident quality, on-call sustainability and operations culture. Teams where knowledge is not stuck in a few senior heads do not just build more resilient systems; they also build more teachable ones.