2019 Invocation Stats Revealed – [Webinar]

Les: So back in December 1999 when the iPhone was nothing more than a dream, the ‘cloud’ was fluffy and full of moisture in the sky and cybersecurity – well that was mostly in Star Wars and Star Trek…

I was ‘seconded’ to a group company that provided Disaster Recovery and Business Continuity services in the UK to help it prepare and be ready to support its customers that fell victim of the millennium bug. That company was called ICM.

So what was the millennium bug that was going to bring the ‘end of the world’, well according to the press anyway?

The issue was simple: back then software and hardware were using two digits for the year instead of four, this would impact all programs using time-based calculations when the digits went from 99 (1999) to 00 (2000). Time would be resetting backwards, not forwards.

So prepared as we could be, we all waited with bated breath for the midnight hour.

Les: Well, nothing happened: but what a memorable New Year’s Eve that was! By why did very little happen? Because of years of preparation and testing that had taken place.

I stayed with the DR and Business Continuity provider as it was proving interesting – I’m still here now… George is going to tell us about the technology changes over the years.

George: In the 20 years leading up to the millennium and Y2K, there was a sea change in the technology available to businesses. From the launch of the IBM PC, the next 20 years saw the introduction of the Public internet, web browsing, the mobile phone and affordable network connectivity.

This opened IT to everyone in the workplace. Office applications became hugely popular with VisiCalc popularising spreadsheets, WordStar word processing and Microsoft Office the office suite.

This drove the decentralisation of IT systems from completely central systems out to the branches, all of which needed to be protected.

George: While there may have been a revolution in the use of IT leading up to the millennium, the same couldn’t be said for protecting IT. Virtually all businesses still protected their systems using tape, a technology 50 years old by then.

While tape capacities had increased to cope with the growth in IT use, its main problems still remain. Backup failures were a regular occurrence – either with the tapes themselves or backups not completing because tapes filled and no one was there to insert the next tape.

Worst, tape failures were not always evident – only being noticed when a restore was attempted – and most of the time, backups weren’t verified.

Physical recoveries were the norm as virtualisation was still in its infancy – so was standardised hardware, so most recoveries were technically complex and not undertaken by the customer.

Les: So in the early years:

• Disaster Recovery back then started as an extension to a maintenance contract (break/fix) to deliver replacement computers to customers who had suffered an event that meant the maintenance contract was invalid so this could be things like fire or flood damage for example
• The service extended to the use of mobile computer rooms
• Back then, less than 30% of customers performed annual DR testing
• Up to the millennium, there weren’t many drivers for business to adopt business continuity, those that existed could be ignored and were largely not enforced

Les: At the turn of the millennium, there were around 14 other organisations in the UK that provided Disaster Recovery & Business Continuity services. Most were providing IT-based services, the Workarea providers were limited and focused around London, Birmingham and Manchester.

We had three Workarea sites with a plan to open up a nationwide network of Workarea sites taking the service to within 1hour of all major cities in the UK.

Doing a DR test was like a ‘jolly boy’s outing’ being a few days out of the normal business watching tape recoveries, stumbling around with little or no preparation. More often than not, the would run out of time and comment, “we’ll fix it next year” hopefully…

As you can see, we were delivering one or two DR Tests each week with on average one IT invocation per week.

Workarea invocations were very rare, on average one every six months or so, in those days it had to be a major business interruption for an extended period before a business would undertake the relocation of its staff and getting services working at the recovery site.

Les: Then, of course, there were the dreadful events of 9-11. This changed how businesses viewed business continuity.

This, with the apparent threat of the millennium bug and corporate failures such as Enron and visible systems failures within the Public sector, all contributed to business continuity becoming adopted as a mainstream business practice for major organisations.

This, in turn, led to the larger organisations looking at the supply chain and the risks that existed.

The consequence of this is was that those smaller organisations, that may not have been previously directly affected by regulation or PLC corporate governance were now facing pressure from their major clients to show proof of continuity as part of general trading terms.

This development was crucial to the growth of mid-market opportunities for businesses such as ours as for the first time it created a driver for organisations to invest in a BC plan as it could be the difference between winning or retaining a client – or not.

Les: So in the years immediately following 9-11, new drivers for adoption of BC planning by UK business were coming in such as:
The Civil Contingencies Act 2004

The Higgs report in 2003 on corporate governance following Enron collapse amongst others

The FSA expansion of scope to mandatory compliance in some financial sectors

BCI and BSI Standards issued for Business Continuity Management leading to BS25999 in 2007

The number of DR or BC providers in the UK continued to reduce with mergers and takeovers taking place

And we at ICM opened up new Workarea Recovery sites across the UK with uptake seeing significant growth in our business and our delivery teams and activity

We were now completing six–eight DR tests each week with rising numbers of Workarea tests – but still low numbers of Workarea invocations- it just took too long to spin up services and move people – most incidents being utility failures but also denial of access becoming more predominant, some due to terror-related incidents

In the main, invocations were still based on the traditional IT DR services with equipment failures of some sort being the main reason customers invoked.

But Disaster Recovery was changing into Business Recovery, and this presented new challenges in how to protect a business, not just technology.

But technology and people recovery were still disparate, very little joined up testing, with low commitment to using Workarea services for short term business interruptions.

George: Backups were still key – but they had now evolved to be stored on disk as virtual tapes or backup sets. This eliminated the major issues with tape – capacity issues were a thing of the past and verification was the norm. Server and disk replication was more commonplace as well, complimenting traditional backups.

The final piece was virtualisation becoming the common server platform, allowing standardised server recovery and the introduction of automation. This led to service tiering becoming the norm – servers and applications could be recovered in related groups, streamlining recoveries.

This not only streamlining their recoveries but also allowed them to be recovered by priority and not by their position on a tape.

Les: Moving on to this time, ICM was purchased by Phoenix and merged with what was then a competitor, NDR. The business increased fourfold now with even greater spread and presence in the UK, the capabilities were extended, infrastructure was upgraded, and services now included holding customers data in our data centre using backup and replication tools.

The number of providers in the UK market was continuing to decline.

All the supply chain and regulatory requirements were now driving business into serious BC at all levels, no longer a tick in the box for the IT manager, now rehearsals were expected to be successful first time and every time… the technical complexity of an IT DR test increased significantly.

As the new combined business, we were now delivering 100 rehearsals a month, supporting over 300 incidents a year and delivering 180 invocations a year, 15% of which were now Workarea as maturity and confidence in the services through testing, was making a difference.

Customers with a pedigree of testing Workarea were more likely to invoke as confidence in the service had grown, and technology had made it simpler and quicker to initiate.

George: It’s hard to believe that cybercrime has always existed. From a few rude words by an American Magician in the early 1900s to being a business in its own right today.

The first recognised modern cyber-attack was in 1988 when the Morris worm infected every computer system in ARPAnet, the forerunner of today’s Internet.

In general terms, until 2010 cybercrime was mainly focused on hacking government secrets – from espionage and civil rights to meeting ET. It was hard to imagine then how quickly cybercrime would develop into the threat that it is today.

George: Rehearsals are now more scenario-based than just testing the recovery of systems, with end-to-end testing now the norm, from the Workarea, through applications to the data centre.

Many companies are now happy to invoke for less than a day, as we can securely connect directly into their environments.

There is strong take-up of new products for online backup and replication that bridge on-prem, hosted and cloud-based services.

This is coupled with an increased focus on cyber-aware protection, with a focus on AV integration and ransomware recoveries.

George: Cybercrime changed around 2010 when Stuxnet was revealed. Suddenly the cyber landscape had changed, and business became a prime focus. While the press still reported on government-focused stories, cybercrime was being industrialised. The successful Email attack on NASA in 2006 showed that anyone was vulnerable.

The rise of zero-day attacks and botnets suddenly became a reality, hitting the mainstream in 2017 with the worldwide WannaCry ransomware attack: Affecting over 150 countries, 200,000 computer systems and costing $ billions to fix. In the UK, over 70,000 devices were affected.

The NHS was surprised by the impact on non-traditional IT systems, affecting MRI, theatre and blood storage systems, to name a few.
Outside the UK, it’s estimated it cost Merck shipping between $600-$800M and FedEx $300M.

While the direct monetary costs are easy to quantify, the indirect people, reputation, productivity costs, etc. are incalculable.
Cybercrime is now big business – for both the good guys and the bad guys.

In less than ten years, cybercrime has evolved to become a normal, daily business threat.

George: In response to the cybercrime threat, we introduced Safe Haven allowing clean recoveries and use of services in a safe, secure environment.

We’ve also seen an uptake in self-recovery, rather than solely assisted recoveries.

This also means that rehearsals can be used by IT to enhance business value and reduce service risk – allowing patching, training, rehearsing changes, etc., to be performed in a safe, sandboxed environment

The rise of public cloud and “as-a-Service” has meant that business’ IT is now a web of on-prem, hosted and public cloud.

Rehearsals and invocations now involve multiple services, organisations, teams and locations. There are now more points of failure than ever before, and IT is no longer a single chain, but a complex web of integrated services.

While technology may be all-pervasive, few business continuity providers are multi-disciplinary, covering Workplace, networking, services, systems, applications, cloud, InfoSec, etc.

Les: Moving on into the recent years and now having been acquired by Daisy, we are completing up to 650 rehearsals a year, supporting 160 incidents a year with Invocations now one a week.

Business Continuity suppliers in the UK are now down to Dasiy and Sungard that can provide true national coverage for UK businesses.

Workarea now accounts for nearly 50% of all customer invocations each year.

Rehearsals are now complex mini projects with desired outcomes and no room for failure from the supplier or the customer.

This is showing our testing activity over the yearsIn the early years, less than 30% of customers completed a DR Test in any one year. Surprisingly that number hasn’t reached any more than 45% in the 20 years I have been gathering these stats.

The volume of rehearsals reduced in recent years due to various contributing factors: the complexity, the wide mix of skills needed and customer commitment.

In 20 years, a total of over 12,000 rehearsal have been delivered.

Now more than 70% of all rehearsals include or are exclusively Workarea.

Les: Over the 20 years, we have supported more than 3,800 customer incidents and responded to nearly 1300 invocations, delivering over 1,000 IT Technical recoveries and nearly 250 Workarea invocations.

The yellow line here shows the % conversion of incidents to invocation; we can see that back in the early days, an IT incident was more likely to turn into an invocation. With the maturity of risk management and BC planning over the years we can see that customers now are more likely to put the service on standby for any potential risk to their BAU activities and so there is a lower conversion rate to actual invocations.

The blue and the green lines show the reducing trend of ITDR invocations due to technology changes and the increasing trend of Workarea invocations over the 20 years.

Workarea incidents and invocations now starting to see the impact of cyber attacks although not all customers either want to talk about being the subject of an attack, or realise they could use the service they have in place to help them.

The fact that cyber attacks are being managed by Information Security teams rather than Business Continuity management could be accountable for the low numbers at the moment, but we are seeing a change in this regard.

George: Prior to 2000, when most incidents were IT failure related, 60% turned to invocation. Following 2000 about 30% of incidents turned into invocations, of these, 50% are not IT failure related.

Sadly there has been a growth in terror-related incidents, such as the 2001 Birmingham bombing and the 2005 London bombings, but these are rare.

2004 saw widespread disruption following a fire in a BT cable duct in Manchester that destroyed 130,000 business and residential phone lines, leaving companies without telephone links, fax lines and access to their networks.

2007 saw a number of major floods, including Tewksbury (pictured). These events led to a significant uptake in Workarea services as business’ lost access to their premises. These significant events aside, most invocation were still IT failure related, with a growing number due to utility and environmental failures in buildings.

Les: If we look at the last full seven-year statistics being the most accurate and relevant for today, we get to see the main reasons why businesses need business continuity.

We have supported some 1,340 incidents over the seven years, with 428 invocations being 32% of the recorded incidents, with 128 being Workarea recovery invocations – some 30% of the total. But let’s look at the changes over that time.

If we compare 2012 to 2018, we can see the changes in the incident types from being IT technology based to more physical and business impacting events. We do get spikes in activity due to specific events, terrorism, extreme weather, wider area events etc.
What we are seeing now is about 30% of incidents turn in to invocations, some 50% of which are not IT failure related.

Les: Let’s look at the last 12 months in detail. The recording of current incidents has been reclassified to reflect today’s threats in the ten categories:

Hardware Failure is now classified along with a software failure including some form of data loss or corruption that is not cyber or virus connected – 23 incidents

Planned Maintenance now includes any pre-planned activity including platform or communications upgrades and physical office moves – these are standby incidents that should it turn to invocation, become one of the other categories – 44 incidents

Environmental Failure now includes flooding caused by a failure of some type to the buildings water supply – 22 incidents

Power Failure is any incident that has caused an interruption to the electrical mains supply to a building including a UPS failure – 30 incidents

Communications Failure is any voice or data or internet access issue causing business interruption – 28 incidents

Adverse Weather now includes flooding due to extreme weather – 2 incidents

Fire can be to the customer’s building or any adjacent building that causes a denial of access to their building – 7 incidents

Denial of Access is any other event that stops a customer having access to their normal site – 1 incident

Civil Unrest or Terrorism can include any other unlawful activity such as theft of computers – 1 incident

Cyber Attack/Virus is any form of cyber activity DDoS etc. – 4 incidents

Les: So the conversion from incident or standby to actual invocation over the 12 months now looks like this:

• Hardware Failure – 12 (33% of invocations)
• Planned Maintenance – 0 as the invocation becomes one of the other categories
• Environmental Failure – 4 (11% of invocations)
• Power Failure – 4 (11% of invocations)
• Communications Failure – 11 (30% of invocations)
• Adverse Weather – 0 invocations in the last 12 months
• Fire – 1 invocation (4% of invocations)
• Denial of Access – 0
• Civil Unrest or Terrorism – 0
• Cyber Attack – 4 (11% of these incidents)
• Workarea invocations and IT invocations now almost a 50 -50 split

Les: The reality is that many of the threats still exist as they always have done. We’re just starting to understand the impact of reported cybercrime, although many incidents are currently misreported.

Unfortunately, the risk of terrorism is still present today, with the official UK threat status still being ‘substantial’, (“attack is highly likely”).

Sadly we’ve seen this rise to ‘critical’ since then on two occasions, both times in 2017 following the Manchester Arena bomb and Finsbury Park mosque attacks.

Communications failures for business’ is increasing, especially with the reliance on access to the internet, the cloud and the rise of the Internet of things (IoT). Now that virtually everything can be connected to the internet, from vehicles, toasters, fridges to more traditional CCTV, the rise in cybercrime is inevitable.

The ongoing power and environmental risks will ever be present – more variable weather patterns also seem to be putting an increasing load on power distribution systems.

The reality is now though, a proven BC plan can be activated and be effective within hours (not days as it was in the early years) and the reluctance to displace personnel to a recovery site is significantly reduced in this era of 24/7 everything.

Les: So what is the future for the industry? From my perspective: we are heading towards integrated cyber and recovery services, services where proactive monitoring and management of customers network and infrastructure becomes part of the business resilience services we deliver, rather than always being reactive to situations we become integral to helping customers avoid many of the reasons to invoke the services.

Providing flexible and adaptable Workarea solutions using the investment made already in our infrastructure to deliver a newer brand of services to meet the new requirements of the modern era where people are always working, and infrastructure is always on. Cybercrime is all around us, most we don’t even get to know about, but organisations need to know there is a service that can work that will maintain business operations in spite of such an attack. We have already delivered a number of these invocations using our Safe Haven product. I expect that we will be delivering many more in the future.

Hopefully, that has given you an insight into our world as a BC provider.