Lessons from the CrowdStrike Incident
Envision IT Pressroom | July 19, 2024
Envision IT Pressroom
July 19, 2024
In the early morning of July 19th, 2024, CrowdStrike released a security content update that caused an estimated 8.5 million Windows-based workstations and servers to simultaneously bluescreen and shut down. Though the defective update hit less than 1% of Windows devices, the damaging effects rippled through several industries interrupting broadcasts, bringing down emergency services including 911 call centers, grounding flights around the globe, halting medical procedures, denying retail transactions, and many other disturbances to daily life. The chaos ranged from simple inconveniences to tragedy. And as people convalesce the human loss, the financial consequences could rise into the billions.
The fix to the CrowdStrike issue was to manually remove a file from safe mode on every impacted device. This is a glacially slow process to recovery which is one of the reasons the widespread outage lingered for days, and many organizations and consumers are still experiencing the effects of the chaos. This specific scenario where a security company’s mistake takes down millions of devices is less common than cyber-attacks, both require a manual effort to reimage workstations in the aftermath. This is why much of our focus and time working with clients involves designing resilient End User Computing (EUC) architectures in consideration of a reliable disaster recovery approach. The CrowdStrike event delivered organizations a taste of what a real cyber-attack is like and is an important prompt to rethink their end user compute strategy. Here are our takeaways:
SaaS-based software requires a functioning endpoint
Many CIOs and CISOs have adopted the strategy for risk mitigation by migrating to SaaS-based applications. This is a natural assumption in that it helps isolate outages and should generally be available in the event of a cyber-attack. What is often missed in the risk calculation of the SaaS-based approach is the critical impact of endpoints – if they are encrypted, users still can’t connect to hosted solutions. The CrowdStrike incident highlights this scenario where cloud-based software was up, but many organizations were dead in the water by non-functioning endpoints.
Thin Clients + VDI facilitate rapid recovery
Envision is a top provider of managed services for Citrix environments. This incident reinforced our daily experience that thin clients + Citrix-delivered desktops enable IT administrators to recover quickly from cyber-attacks and botched updates. On average across hundreds of environments, we were able to restore Citrix desktops to thin clients in less than 30 minutes for each of our client organizations that had CrowdStrike installed. This is consistent with our experience in ransomware recovery efforts for organizations. Repeatedly recovering Citrix + thin client is orders of magnitudes faster than individually reimaging PCs. For this incident, the difference in recovery time was minutes vs days for many organizations.
Managing third-party risk is difficult
To combat cyber security threats, organizations have become experts at quickly deploying third-party patches from trusted partners. In many cases, these updates occur automatically as was the case with the CrowdStrike content updates. Balancing risk with expedient updates and comprehensive testing is challenging but made easier with Citrix provisioning technologies. By disabling automatic updates and thoroughly testing with a User Acceptance Testing process, organizations dramatically reduce the risk that bad updates cause wide-spread outages such as the CrowdStrike debacle. Moreover, you can guarantee 100% patch compliance with Citrix-based desktops since they are all created from a single image. This allows IT administrators to fully test all scenarios which reduces downtime.
In conclusion, the adoption of thin clients and Citrix virtual desktops stands as a beacon of resilience in the face of the ever-evolving threat of ransomware, faulty application updates, and other numerous issues affecting end points. The CrowdStrike melee underscores the necessity for resilient EUC architectures, and VDI emerges as a consistently reliable solution. By compartmentalizing desktop environments and minimizing the potential impact of breaches, VDI not only fortifies an organization’s defenses but also empowers them to maintain operational integrity amidst cyber adversities. It is a testament to the foresight of embracing VDI as a cornerstone of risk mitigation strategies, ensuring that businesses remain unyielding in the battle against digital threats.
