hcg
5
Apr

Easter Sunday- What alerts did you receive?

   Posted by: jasoncbunn   in IT Leadership

Today we celebrate Easter Sunday, when Christians celebrate Christ’s rising from the dead.  This should be a relatively quiet day for IT.  Depending on your business, it even may be the end of a three-day weekend with the offices closed for Good Friday.

In the database administration space, Sunday is our maintenance day, at least for our firm.  We actually recently expanded our maintenance window to encompass most of the weekend in a staggered fashion.  We used to launch our integrity checks all at once and that time quickly became very well known by the storage team due to the IO load on the SAN.  Thus, we fanned those out over most of Saturday since many of the applications are offline over the weekend.  Sunday is reindex/reorganize day and even that has become much more spread out across the course of the day.

If all goes well, we receive zero alerts.  No calls, no pages.  I’m always perturbed when we get a page or alert for something that is expected or typical.  If the response is, “oh yeah, that happens.  Just ignore it,” then why are we getting called, especially overnight or off hours?  If I’m going to ask my team to rouse themselves out of bed, to remotely connect to our systems, and read an error message, the very strong preference is that there is an actionable item that needs to be corrected at the other end of the message.  If it’s a benign configuration anomaly, or a status message, then send an email and wait until morning.  If the process normally retries successfully, then email on the first failure, retry, and let my team sleep unless it fails a second time.  I don’t shy away from the work– if there is work to be done, certainly wake us up.  But the ideal state is that we only use the immediate channel of a page, text, or call if there is action to be taken.

So, we expect a quiet Easter and will pick things up on Monday.  Happy Easter!

This entry was posted on Sunday, April 5th, 2015 at 11:46 am and is filed under IT Leadership. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a reply

Name (*)
Mail (will not be published) (*)
URI
Comment