Mail Server Failure

At approximately 4:30am PST our mail server suffered an unexpected failure.  We have isolated the problem to a corruption of the mail database.  We're taking steps at present to recover the database from the available log files.  By 11am we will know if that process was successful.  If so, we will be fully operational at that point.  If our database fails to recover, we'll initiate a full restore from last evening's backup.  That could take up to two more hours to process meaning, worst-case, we will be back online at 1:00pm PST.

Sorry for the inconvenience.

-Alex

----

Update: Our attempt to run the recovery utility is being hindered by a lack of sufficient drive space.  We're exploring the possibility of running the recovery process on a different system and then moving the repaired database back to our mail server.  I'm still confident in our 1:00pm PST repair time.

We also discovered that our latest backup is from Friday evening rather than Sunday evening.  I still hold full database recovery as an option but I'm going to try to avoid it as we would lose all the mail that came in over the weekend.

-Alex

----

Update: Chris successfully got  Daryl to act as a surrogate recovery server.  We're waiting for results of the process now.

-Alex

----

Update: Take a few moments to go through outlook and your phone. Make a note of any important messages that you received over the weekend.  There is a chance that the recovery process may delete some of those messages from your inbox.

----

Update: The recovery process is chugging along but it is taking dramatically longer than I expected.  At present speed I'm estimating that we won't be functional until tomorrow at noon. At this point there is a choice:

1) continue the recovery process, recover all mail, but have limited access to new mail until tomorrow at noon.
2) restore from the Friday night backup, lose all mail between Friday at 8pm and Monday at 4am but have full access to new mail by 2pm today.

feel free to email me with your thoughts but I'm going to make this decision in the next few minutes.

-Alex

----

Update: I've decided that the productivity lost today is more valuable than the minimal number of messages we may have received over the weekend.  I asked Chris to cancel the database repair process and I'm recovering from our Friday evening backup. ****YOU WILL LOSE MESSAGES THAT CAME IN AFTER FRIDAY AT 8:00PM**** Right now, you should go into your phone, ipad, or outlook and make note of any important messages you received over the weekend.  I estimate that we'll be live again at 2pm today.

----

Update: Estimated recovery time now 2:30pm today.  Accuracy confidence is 90%.  The data recovery job had a slight hiccup but seems to be going smoothly now. 

----

Update: Mail server is now functional and stepping through it's queue of cached messages.  I have successfully sent and received mail and can confirm that we are fully operational.  There will be a rescan of your inbox at 2:30 to remove all the spam that is currently bombarding your inbox.  If necessary I'll do another rescan at 3:00.

Again, sorry for the inconvenience.  Be aware that messages sent to you between Friday at 8:00pm and Monday at 4:00am are most likely lost.  If you expected a message in that time frame, let your constituents know about the failure.  We have an option to recover that data if it becomes necessary but it is a lengthy process. If you believe there may be crucial emails in that window, please let me know.

-Alex