Post Production Stress

Two weeks before we went live with one of our products. A product where I worked personally for 4 months. I still do not feel I did justice to the product as the actual problems still glare at me when I see the application. It was a whole lot of work within 4 months and a more work when it went live. As the product was the first one to go live from us, we where a little tense. Some glitches here and there and I was all stressed out.

Its a whole big deal making an application live or into production. Especially if the application is not owned completely by you. The task of closely monitoring it, making sure that nothing breaks, maintaining the backups, ensuring that binding is done quickly and effectively if something breaks is completely done by the team. This is very tense. And I deeply appreciate all those managers who sympathize and stay beside their team during those stressful times. If you are answerable then you better have pretty good answers.

The build technique that was made is my main area of concentration this time around. Indraneel was the person in charge and I sat with him one time to deploy the code live. The scripts written via capistrano were excellent. The deployment can be to the test server or the live server. This requires issuing of different commands and writing separate tasks. The commands issued are remote commands and they usually are stop and start commands for mongrel. This application had also ap4r in it for asynchronous processing and therefore required a restart for that as well.

Monit has also been installed in this server and ap4r and mongrel has been configured to restart once stopped. The mod proxy balancer has been put along with apache so that mongrel clustering can be implemented. The balancer member written inside the proxy tags has an ip associated to it. This is where the actual transition in case of updates for the live application comes into picture. Using apache we could have something called a graceful restart using the 'apachectl graceful' command. Normally when a new code is deployed with all those changes and upgrades the server (mongrel in this case) must be restarted. But with graceful restart Indraneel just has to do minimal stuff. He first deploys the new code onto a server started with a different port, say mirror. Now what he does is change the balancer member port number to the mirror port and does a graceful restart. And wallah .. a work well done. This could also be automated if there were 2 virtual host files and according to the live/mirror deployment, the file is overwritten by the specific files.

Great work !!