Where Testing Ends and Real Life Takes Over

It is simple…

When working on mission-critical applications, some are high profile enough to make the news if they go wrong.

This just happened to be one of those times. Given the right conditions, toppling this system without comprehensive backup plans could end with you being the breaking news. (Wordplay intended.)

This was our second attempt at go-live. The first one had to be rolled back.

We had spent months testing, validating and rehearsing. Users were trained, and we were confident they were equipped with the operational knowledge to switch to the new system.

Or so we thought.

When deploying a system, a maintenance window is usually required. This is a period of time when there is little to no system usage by users. In this case, it was scheduled between midnight and 3 am Not the best time to be solving complex problems if you are not a night owl, but we made do.

We went live.

Everything was flowing. The system behaved as designed, and the users were excited to play with the shiny new toy, which made their work more seamless.

Until one user decided to enhance the system further.

In order to contact mobile app users, their radio numbers had been stored in a separate system. This integration was never part of the original scope.

However, the user believed it would be an even better experience if all the information were in one place, removing the need to switch between applications.

Their solution: amend the name field by adding the radio number as a suffix.

The field was editable with the right access, and this user had that access.

They began updating names.

On the surface, they were improving the experience. But under the hood, every update was quietly breaking the integration link with the mobile application.

It only took a few minutes before the panicked calls started flooding the support team.

The mobile app had stopped updating tasks.

Troubleshooting led us to the user updates, and quickly reversing them cleared up the issues.

This was a textbook edge case.

We had assumed users would follow a particular workflow. But user-need got the best of our predictions.

And to be frank, that user was not off-script, they were writing their own.

It goes to show:

Every user’s fringe action should be a design review.