Importance of rollback testing in migration


Hi guys, have been busy with a project, hence the lack of posting.

I have been tasked to migrate a current system that relies on Windows server (yeah everyone hates windows) to one based on Linux. The servers received signals from a particular car device in TCP format.

The challenge here is to make sure when you point the car device from windows server to linux server, the user can’t feel it, as it is already in production. So we do have a few prototypes in place. We divert the signal from the windows to linux and modify the api to be able to read from 2 different data sources. Then we start to redirect some customers car devices from sending signal to the windows server to sending it to linux server. And lo and behold, customer complained that one of their feature is not working. So we decide to switch back. Then I realised that the way how the engineers do the reverse pointing is to send the signal back to the device via the server using TCP, identified by the ID. Oops, didn’t implement this into my new system. Lucky there is an alternative, which is to use an SMS to the device SIM card.

Hence a lesson learned here. Don’t just test the new system and migrate. If the current system is working but unable to scale and collapse after a certain load, and you have set up a new system that is tested to be able to handle that load, but not battle harden, it is a must to do rollback tests so that when something went wrong, you know you fall back to the old battle harden system before customers start ringing. Don’t just migrate. Do a tango and roll back as well to test if the system work as normal, then you start to go all out to shift one system to the other.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s