Please refresh this page regularly
---
Symptoms:
When using Microsoft Outlook, the program will either consistently ask for password confirmation or display as "disconnected" or "limited availability" or something similar.
or
When using Outlook Web Access, you will be unable to login and the site will consistently prompt "username and password incorrect" or something similar.
---
Dear Customer,
We have been informed of a major SAN, (storage area network) failure, at one of our partner datacentres. This specific datacentre in Park Royal, London, physically host some of the Microsoft Exchange mailboxes, which we manage and provide to our clients.
At present we believe approximately 8,500 exchange mailboxes are effected. We are being informed that level 2 and 3 Exchange engineers are present on site, including vendor engineers from HP and also consulting engineers.
The datacentre are treating this as a 'Priority 1' incident and have been working through the weekend, including nights to ensure that this issue is resolved ASAP.
We will post any updates that we receive as and when here. We thank you for your continued patience.
UPDATE 29/09/14 - 08:45
We are being told that the SAN is now back online, fully operational and responding as normal. Engineers have been working since the early hours to bring mailboxes online cluster by cluster. We expect to have returned to 90% of normal service by 09:00 GMT.
UPDATE 29/09/14 - 09:20
We are being told that the datacentre are currently attempting to bring the remaining three clusters back online. They are unable to give a precise ETA for when the remaining clusters will be accessible but they expect material progress before 10:20.
They are aware that mail delivery is severely delayed and that inbound and outbound mail may not function normally or will have extended delay for some time - we anticipate that queued mail will take a few hours to clear, during which time email delivery performance will gradually improve.
A further update will be provided by engineers at approximately 10:30.
UPDATE 29/09/14 - 10:50
We have been provided no further information on when we will be able to restore access to the remaining mailboxes. We expect that access will be restored over the course of the day.
The datacentre have currently raised an urgent support request with Microsoft regarding three of the main clusters affected which sum approximately 3000 mailboxes. Microsoft will be working with the DC engineers to restore access as normal.
We are very sorry that we do not have any more concrete information and will provide another update as soon as possible.
UPDATE 29/09/14 - 12:44
We are being told there will be an email update sent out very shortly. As soon as we have any details, we'll post them here. Thank you for your continued patience.
UPDATE 29/09/14 - 14:27
We are chasing an update from the engineers on site who are busy resolving the problem. We apologise for the lack of information and we'll try and update you shortly.
UPDATE 29/09/14 - 15:59
Engineers in Park Royal are working with their partner Microsoft on restoring access to two main clusters, representing around 2500 mailboxes in total. They now have cluster and core services functioning on these two remaining key clusters and are now restoring databases. The percentage of available mailboxes will continuously increase as we restore databases on clusters EXVS01 and EXVS02.
They understand the frustration and a need for a clear ETA, however, at this stage it is challenging to predict. They are continuing to restore services and we will update you on an approximately hourly basis until this is resolved.
UPDATE 29/09/14 - 16:50
Further to our earlier update database restores are continuing. More mailboxes are now available and we would also like to report that mail flow is continuing and the edge server queues are clearing.
We will receive a further update shortly after 17:00 GMT.
UPDATE 30/09/14 - 09:00
Datacentre engineers have been working on the restoration of the exchange clusters overnight. At the moment over 70% of the mailboxes are up.
The restoration of one cluster has been completed and the restoration of six other clusters is in progress at the moment. Continuously more mailboxes will come up this morning as the databases are restored. They are still experiencing issues with two clusters and are working closely with the vendor to resolve these so that restoration of normal service can be completed.
Our apologies for the lack of further information. The datacentre are still treating this as a critical priority issue as it is effecting many customers using Exchange across the UK. We will provide further updates here as soon as we receive them.
If you are effected by the mail store outage, once your mailbox is mounted by the cluster, it will come online automatically for you.
UPDATE 30/09/14 - 10:31
Further to our earlier update, we have been informed that nearly 75% of the mailboxes are up and accessible – over 6250 out of 8500. The database restores are continuing. Progressively more mailboxes will come up this morning as the databases are restored. The datacentre team is working closely with HP to resolve issues on two clusters.
UPDATE 30/09/14 - 16:30
The database restores are continuing, engineers have confirmed they are experiencing high latency, which has slowed down the recovery process. We will share further information in two hours or earlier if there is a significant update.
UPDATE 01/10/14 - 09:30
We are being told engineers are still experiencing a problem with the restoration of the final clusters. We are therefore proceeding to create new Exchange Mailboxes for all users effected. Users will then be able to send/receive from a new mailbox, using Outlook Web Access whilst the ongoing problem is resolved. Once solved, the plan is to perform a merge operation to get the historical email back into the new mailboxes.
We are creating the mailboxes now and working with all users effected. Thanks for your continued patience.
UPDATE 02/10/14 - 10:13
Yesterday we successfully created new mailboxes for any users still effected by the above outage.
The engineers at the datacentre have told us that 90% of mailboxes are now online and there are only two clusters that remain effected by the outage. We have been informed by the DC engineers that HP have ordered two new disk arrays to site which will allow them to fully complete the restoration process and return to "business as usual". RatwareUK intend then to migrate and merge data from this legacy platform into the newly created mailboxes.
We will provide any updates here ASAP. Our continued apologies for any inconvenience caused.
0 Comments