My AX 2012 Upgrade Project Plan - Deployment and Operations phase - Deployment Checklist
So, you did really good planning, you completed your upgrade iterations, you tested and tested and tested, and you now have business approval to go live ...
Time to develop your Go-Live checklist!
For us, the release checklist was a huge success factor. We did all our test upgrades during UAT on weekends, and our checklist was what kept a distributed team not only on track, but engaged and informed in detail about each task in the checklist.
Our checklist was not what you would imagine in terms of the checklist in the upgrade cockpit in AX. Our checklist had to coordinate every task above and beyond the tasks in AX. That included coordinating with multiple people not from our upgrade team including operations, help desk, security, external entities with integration points and the business.
For each day of the weekend release, we had individual and granular tasks with times and responsible team members. Our very first checklist for our first upgrade iteration had 22 items, our final Go Live Checklist had 127 tasks!
I have broken down the images of the checklist as it is too long to be readable as a single screenshot. This is for your inspiration, I am sure your context is different, and you may not need all of the steps we have, or you may need many more. In any case, first, here are the days up to the release weekend:
Friday was the big readiness day for all, and that night, the main technical part of the upgrade was done:
And this is what our war room looked like on release Friday end of day:
Ready for Go Live! |
Yup, we were really prepared, so no one needed to work late, stay over the weekend etc. Remember we did 9 weeks of parallel testing where we completed 7 full upgrades, so this was a process we really had practiced and knew all the details. If you notice on the checklist, we had expected times for each task, and we were quite close to these as they were repeatable and predictable, so we were off early on that Friday of our AX 2012 Upgrade Go Live weekend. We all had a backup plan to all show up in the office should there be a need, but the weekend went quite smoothly, so we all worked on the upgrade from our homes.
Saturday was about validating and testing the upgrade:
Finally, Sunday was all about integration testing and configuration as we got ready for Monday:
Done!
So, how was Monday? Well, AX crashed right after lunch! We discovered a memory leak issue, and had to restart servers at lunch and before the start of the next day (And we had 96 MB RAM on each AOS!!!). Basically, we had to collaborate with Microsoft for about 2 weeks to get a fix. This was our biggest surprise as this never happened during UAT. The reason was, that all our parallel testing days were half days, so we never tested long enough to reach the critical memory issue that caused AX to crash! And after we applied fixes for the day, we restarted the boxes ...
Other than that, we had initial performance issues, some user issues, and our stabilization period lasted for 6 weeks, 3 tough ones due in part to the memory leak issue, then a couple of light ones and the last week of stabilization was just to make sure we could declare the upgrade completed to the steering committee.
A great thing we noticed, is that due to the nature of our parallel testing, our users did not experience productivity loss in the initial weeks of go live as they knew the new system very well, it's like we had always had it! And, in addition, those tough 9 testing weeks also served as quite hard re-training of all our processes so we had very low process performance impact during stabilization.
The following months after stabilization were focused on performance. We worked with Microsoft to conduct a performance health check, which had us document all our key processes and document traces of the AX code execution, data base traces, server configuration audits etc. We got a report back with recommendations for each area and we implemented all. We are quite optimized today.
In addition, we had one of the AX performance team leads come visit us for a week to teach our team all the performance techniques and tools for AX 2012. I recommend you do the same!
Ok, we are done with the story of our AX 2012 Upgrade! Thanks so much for reading and I hope you were able to get a few good golden nuggets for your project.
Next time, I am going to share with you our thoughts about where we are going next, and also share with you our thoughts around "Upgrade Fatigue" as we also upgraded to R2 a year after, which was a medium sized upgrade and then short after did the CU7 update which felt like a little Upgrade. We decided to not go R3 and actually wait for the next version, which we hope will be all cloud.
Nice post! Thanks for sharing this, Andre. :-)
ReplyDeleteThat sounds like an amazing team. :)
ReplyDeleteUlualensiopa Tiffany Martinez https://wakelet.com/wake/Y_6bAbSP0ByodoTe7OHeX
ReplyDeleteuntirekvau