Friday 02 December 2022
States of Guernsey services have been stable overnight, gov.gg and other States websites were restored last night and the majority of internal systems are working.
There continues to be issues with some internal systems linked to individual services. WiFi has been restored in the schools.
Critical systems within the blue light services have been maintained throughout the disruption and patient care in HSC has not been affected.
Following widespread disruption to States IT systems this week, caused by the failure of air conditioning systems in the server room on Friday, in recognition that the community understandably has questions around this week's challenges we are able to provide the following details based on what we know at this stage - noting that further investigations are ongoing but the priority is understandably getting systems fully back up.
- The States of Guernsey has a main Equipment Room (which has circa 500 servers running) housed at Sir Charles Frossard House.
- A back-up Equipment Room is housed at Edward T Wheadon House.
- The live data is replicated across both sites.
- The States also does an incremental back up of all data every evening and weekly full back up to a third site.
- A warning system is in place to alert the States if the equipment room gets too hot. It is set to 25 degrees and alerted us there was an issue on Friday morning (25th November 2022).
- IT engineers arrived at the site within 20 minutes, confirmed the air conditioning systems had failed and the room temperature was at 44.1 degrees so immediately carried out health checks on all equipment.
- Third-party contractor were notified immediately and dispatched cooling engineers to site.
- The room temperature had increased to 48 degrees upon the arrival of the third-party contractor.
- The main system went into 'preservation mode', which is a safety mechanism for it to automatically shut down and protect itself and the live data in the system.
- This successfully safeguarded all live data held by the States of Guernsey, however it did cause significant performance issues across the States IT network.
- However, despite a 'fail over' being attempted, switchover of the services to the back-up equipment room at Edward T Wheadon House failed.
- It is unclear at this stage why the main equipment room did not successfully switch to the back-up and this requires further investigation, which are underway.
- Further information will be provided to the public once those investigations have been completed.
Deputy Peter Ferbrache, President of the Policy & Resources Committee, said:
'Clearly the circumstances of the last week are not good enough, despite the significant amount of work from engineers who have been trying to get systems back up and staff across services managing any disruption. Being a week in and not fully operational is concerning and we need answers as quickly as possible. Our Committee, States Members and the wider community understandably want answers in language we can understand, and importantly reassurance about what will happen to make sure we never see a repeat. We will be expecting those answers after things are back to normal.'
Mark de Garis, States of Guernsey Head of the Public Service, said:
'An outage such as we have experienced over the last week is unacceptable and urgent actions are being taken to fully understand how this happened. The States of Guernsey runs a large number of systems supporting a huge range of services; for example we have bespoke systems in areas such as children's services, the ports, Beau Sejour, social security systems responsible for the payment of benefits and many online services through gov.gg. This makes the recovery of systems extremely complex. Agilisys engineers and third-party contractors have worked tirelessly over the last eight days to restore services as quickly as humanly possible and I would like to publicly thank them. This however remains a completely unacceptable situation and I want to apologise to all our service users who have been affected by the disruption. We will provide further information as soon as the detailed investigations are complete.'