We run a number of production servers, some of them using IIS and ASP.NET, and of course they run around the clock. We have used various monitoring tools and services throughout the years we’ve been doing this, but in addition to the mounting expense of adding more and more servers to our monitoring services, and upgrading to shorter and shorter monitoring intervals, we were still left with two issues.
First, regardless of using our services most frequent testing intervals we could sometimes be down for longer than we liked — in part because an outage could happen at an inopportune moment, when we were away from a computer for example.
Second, it’s no fun being awakened in the middle of the night just to restart IIS. And with customers in every timezone around the globe, any outage anytime disrupts someone trying to get their job done.
We’ve gotten good at what we do — we’ve been at it for more than a decade, and outages are few and far between for stable, mature products. But we absolutely hated any outages that although unavoidable, would last for more than seconds.
How did we address this? In two steps. First, we wrote mobile tools for the manual restart. Then we automated the whole thing.
Although many of us tote iOS devices we all have Android phones, so an Android app was the obvious starting point. Coupled to our own non-IIS, custom port web server, the app allows us to check the health, and restart as needed, any of our IIS / ASP.NET servers (let’s face facts, as nice as IIS and ASP.NET are, they fall over more often than Apache or Nginx on Linux).
Actually the solution we developed allows us to execute any Windows command we want, with all commands both centralized in a “hub” server and externalized from the code so any command can be added to a simple configuration file and it’s added to the available options.
This solution meant when we got the monitoring service alert for a particular server, all we had to do was tap-tap-tap on our smartphone and problem solved. This meant faster responses to problems. That might mean waking up in the middle of the night and using the smartphone by the bed to fix it in seconds before rolling over and going back to sleep, or it might mean pulling over to the side of the road to take care of it instead of having to call someone else or hurry to the office or home to be in front of a computer.
The second step was to enhance our “hub” to do its own monitoring and have it issue the command to fix the problem itself. This had several advantages. First, we could monitor on an even shorter interval at no extra cost. Actually, even with a monitoring service as a backup we’re saving money by choosing a less-costly interval. Second, automating the restart means quicker response, which means beter service for our customers. Third, no wake up calls in the middle of the night.
Is this something YOU need? Because we’re considering making it into a product. Give us your feedback in the comments or by emailing us at info at mdotech.com.
{ 0 comments }
