On January 26th 2023, a minor release was deployed to all environments containing mainly bug fixes and a code clean-up. Release 4.1.4 was deployed to the US and CA regions between 10:34 and 10:48 CET and to the EU region between 17:37 and 18:02
After the release to the EU region, a number of automated sanity checks were failing on some environments, and we started investigating the cause. The sanity checks have some flaws and as the majority of the environments were passing the tests, we didn’t see an immediate reason for concern.
Our support agents reported that a number of test takers received an error page (“please refresh after 3 minutes”) when trying to run the system check or entering the exam setup for their exam.
In the meantime, analysis of the failing sanity checks showed also that some of these tests were failing with the 3-minute error page, which prompted us to rollback the release. Starting at 21:20, environments were being rolled back.
At 21:57, the release was rolled back, and all environment were running on the previous release.
Impact: Students on institutes with custom translation files were unable to start the system check or the exam. 5 customers
Lead-up: Deployment of release 4.1.4 containing a code refactor with a bug.
Resolution: We rolled back the release which immediately fixed the issue. Later investigation of the problem found the root cause. We have remediated the bug by fixing the code causing the issue, updated our tests to better cover this scenario and will be updating our end-to-end tests to cover the custom language scenario. We’re also improving our code review procedure for this kind of situation (code clean-ups).