[sf-lug] Sonic email (etc.) booboo

[Michael Paoli]

"
IMAP Server Issues post Update/Rollback
September 21, 2018

Update 16:19:00 PDT: The status of our email system is continuing to
improve - we will post an update as our systems return to normal
operation.

Update 13:00:56 PDT:  In order to allow the cluster to recover we've
taken steps to reduce the total number of user who can log in at once.
This appears to be having positive results and is allowing the storage
pools to catch up with the load.  We still don't have an ETA but the
situation is improving.

Update: We believe the underlying issues have been been resolved.
However, it will take some time for IMAP and POP services to return to
normal.  The fix has resulted in substantially increased IO load on our
mail spool storage as all mailbox indexes have to be rebuilt.  This
rebuild process can take a long time to complete, especially for large
mailboxes, and it will be some time before the process is complete for
all active mailboxes.

Update: We're continuing to restore services for effected users.
Please note this may also affect some Fusion DSL user's voicemail
services.

Update: mailboxes for many users have been fixed and are working
properly.  We're continuing to repair remaining effected users.

Unfortunately, the mail server upgrade and rollback has caused issues
for our IMAP cluster and it looks like most IMAP sessions are failing.
We're working to restore services ASAP and will have more information
forthcoming.  -Kelsey and Grant.
"

https://sonicstatus.com/