AWS/Microsoft Outage Causes Email Disruption

Posted on
Microsoft email down 9-29-20

 

Amazon + Microsoft failures =

The Great & Powerful Amazon and Microsoft can fail!

Combating this “double whammy” we have put in several 24 hour days

some know as we have spoken to them at all hours.

No complaints, this is what we have been doing since 1992. Over 29 years and only a handful of issues that you can count on 3 fingers.

  • Yes – Within 24 hours almost 95% of systems were back
  • Yes – We know if you were in the 5% it is a problem
  • Yes – Some issues took time for them to fix: some phones not 100% syncing and a few reporting a 30-60 day gap in past emails – most all have been solved – if u see a gap in emails let us know, we are working on these as a case by case basis.
  • Yes – We had backups and some were corrupted
  • Yes – Most if not all customers were operational in a few hours.

What we must remember, and the lesson learned is, “technology scaled at this level can fail”.

What you may be asking is “How can we protect from this in the future?”

The short answer is “You cannot”

Failures will always occur and data loss is always a possibility – If you have not lost data than you have not been using computers long enough.

The longer answer is 2 fold:

  • You can pay for email backup services or manually backup your emails.  Just like remote backup protects your data, email backups can protect your emails – or at lease add another layer of protection (remember, the mighty Microsoft failed)
  • You can back up your own emails on your system – you can do this daily, weekly your choice. It is a manual process and requires you to physically do it – if you forget for a week/month than any recovery will be back to that time of last backup. If your pc failes than your backup is also gone. This has always been available as one of the hundreds of features of outlook.

Read: How to Back Up Outlook: https://www.bulldogtechinc.com/2015/07/06/how-to-properly-back-up-microsoft-outlook/

Bulldog Tech always offered email backups at an additional cost  – “why pay for a service that is rarely used, over 10 years we have never lost emails” you ask and its a good point – Until something fails.

Moving Forward – Adapt and Overcome

  • We added a 3rd data center in the Midwest to handle load and fail-over.
  • We will be migrating to the newest Microsoft Exchange versions as Microsoft is NOW forcing an upgrade. Once completed, will have the additional features offered by these upgrades which we will be outlining as we move forward with the migration.
  • We will be reducing prices on email backups for those customers that wish to add another layer of protection

We will be sending out update emails as we move forward on these.

As always thanks – if you have any issues, email, and call us – we are still here!

Anthony and the vanguard staff!

From Amazon tech support:

“The service team was not able to recover the impaired data volume  This happened due to an underling host issue.” Please note, that we make every commercially available effort to ensure Amazon Web Services is highly available and resilient, however hardware and software failures are unfortunately inevitable in technology at this scale. On behalf of Amazon, please accept my apologies”

Microsoft:

Microsoft Outlook was down as problems with Exchange Online spread worldwide. Microsoft Outlook has suffered a worldwide outage, the second within a week for Office users. On Monday, September 28, Microsoft users suffered a massive outage impacting Teams, Office 365 and Outlook. Now, just 48 hours after you thought everything was back to normal, Microsoft Outlook has gone down again. Within an hour is was confirmed that the issue affects users worldwide.

Microsoft confirmed that users might be experiencing problems with the Exchange Online protocols, including Outlook desktop and mobile.

Read the entire article:

https://www.forbes.com/sites/daveywinder/2020/10/01/new-worldwide-microsoft-outage-confirmed-heres-what-we-know/#354ffd85d370