Module 4: Troubleshooting, incidents, and recovery

seized up

To stop working suddenly and completely.

The entire checkout pipeline ... ... during the Black Friday traffic spike. [Past]

fell through

(Of a plan or fix) to fail before it can succeed.

The last-minute workaround ... ... when it introduced a new regression. [Past]

mounting up

To accumulate steadily until the total becomes hard to manage.

With the API down for three hours, support tickets were ... ... fast. [NB: Continuous]

zeroed in on

To focus attention precisely on the most likely cause or target.

The team quickly ... ... ... the misconfigured load balancer. [Past]

ruled out

To eliminate something as a possibility based on evidence.

Early diagnostics ... ... a hardware fault, so the team looked at the application layer. [Past]

narrowed down

To reduce a large set of possibilities to a smaller, more focused group.

After an hour of log analysis, they had ... ... the causes ... three candidate services. [Past]

pin down

To identify something precisely after investigation.

It took two engineers most of the night to ... ... the exact line of code responsible.

roll back

To revert a deployment or config change to a previous stable state.

The team decided to ... ... the release as soon as error rates started climbing.

bottomed out

To reach the lowest point of a decline before recovering.

Traffic ... ... at around 3 AM and slowly began to recover after the rollback. [Past]

stripped out

To remove something completely rather than repair or work around it.

The engineers ... ... all corrupted records and reprocessed the data from scratch. [Past]

written off

To accept that something is lost and stop trying to recover it.

A small number of failed transactions were ... ... as unrecoverable. [NB: Passive Voice]

bounced back

To recover and return to normal after a failure or setback.

The service ... ... within hours of the rollback, well before the morning peak. [Past]