VMware Patches for Spectre

After releasing the initial security advisory VMSA-2018-0002 to discuss Meltdown and Spectre vulnerabilities, VMware released yesterday the second advisory on the matter – VMSA-2018-0004 – VMware vSphere, Workstation and Fusion updates add Hypervisor-Assisted Guest Remediation for speculative execution issue.

VMSA-2018-0004 – Hypervisor-Assisted Guest Remediation

Updates of vCenter Server, ESXi, Workstation and Fusion virtualize the new speculative-execution control mechanism for virtual machines. As a result, a patched guest operating system can remediate the Branch Target Injection issue (CVE identifier CVE-2017-5715). This issue may allow for information disclosure between processes within the VM.

Affected VMware products:

• vCenter Server 5.5, 6.0, 6.5
• ESXi 5.5, 6.0, 6.5
• Workstation 12.x (patch planned; update to 12.5.9), 14.x (update to 14.1.1)
• Fusion 8.x (update to 8.5.10), 10.x (update to 10.1.1)

Before Patching – Performance and Capacity Management

Before starting the patching process, you need to acknowledge and to plan for the effects of a potential issue with implications on capacity management: the fixes for Spectre Variant 2 (CVE-2017-5715) may bring a CPU performance penalty. It’s hard to estimate the performance impact, but what you can do is to assess your current CPU utilization. You can also prepare contingency plans if your CPU utilization will go through the roof (such as standby servers or a list of non-critical VMs which you can power off). Microsoft published “Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems”. Here is the summary of Microsoft’s findings:

• With Windows 10 on newer silicon (2016-era PCs with Skylake, Kabylake or newer CPU), benchmarks show single-digit slowdowns, but we don’t expect most users to notice a change because these percentages are reflected in milliseconds.
• With Windows 10 on older silicon (2015-era PCs with Haswell or older CPU), some benchmarks show more significant slowdowns, and we expect that some users will notice a decrease in system performance.
• With Windows 8 and Windows 7 on older silicon (2015-era PCs with Haswell or older CPU), we expect most users to notice a decrease in system performance.
• Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance. This is why you want to be careful to evaluate the risk of untrusted code for each Windows Server instance, and balance the security versus performance tradeoff for your environment.

Patch Your vCenter/ESXi Infrastructure against Spectre Vulnerability

Patching vCenter infrastructure against speculative execution vulnerabilities is a case similar with the scenario where you want to enable MTU 9000 in your network: you have to do it end-to-end, for each object involved along the way. Back to Meltdown/Spectre vulnerabilities, you need to patch BIOS of the physical ESXi server, vCenter Server, ESXi, antivirus and the operating system of all virtual machines.

For BIOS patching you need to contact your hardware vendor. Some of the vendors already started to publish new BIOS version (such as Dell, Lenovo, HP). Even if some of the microcode patches may already be presented in ESXi patches, VMware strongly recommends to apply BIOS/firmware update provided by the hardware vendors.

In regards to VMware products, as usual you have to first update vCenter Server. VMware released on 9th January 2018 new versions for all supported vCenter Servers:

• vCenter Server 5.5 – update to 5.5 Update 3g
• vCenter Server 6.0 – update to 6.0 Update 3d
• vCenter Server 6.5 – update to 6.5 Update 1e

All the above vCenter patches introduce a potential issue with Enhanced vMotion Cluster. At some point, you will have both patched and unpatched ESXi servers in the same EVC enabled cluster. You cannot vMotion a VM from a patched ESXi to an unpatched ESXi. After you patch all ESXi servers from the cluster, the cluster automatically upgrade its capabilities to expose the new CPU features introduced by the patches. You will no longer be able to add unpatched ESXi servers into the patched cluster.

Second step is to patch the ESXi:

• ESXi 5.5 – apply patches ESXi550-201801401-BG and ESXi550-201801401-BG
• ESXi 6.0 – apply patches ESXi600-201711101-SG, ESXi600-201801401-BG and ESXi600-201801402-BG
• ESXi 6.5 – apply patches ESXi650-201712101-SG, ESXi650-201801401-BG and ESXi650-201801402-BG (Update 14 January – Based on Intel information, VMware puts a hold on recommendation for all the patches associated with VMSA-2018-0004. Based on same information, hardware vendors like Dell and HPE retired some of the BIOS patches.) 

Down to VM level, you need to make sure all virtual machines use Hardware Version 9 or higher. Virtual Hardware Version 9 is a minimum requirement for Hypervisor-Assisted Guest Mitigation, however for best performance VMware recommends Virtual Hardware Version 11. Version 11 enables Process-context identifiers (PCID) and Invalidate Process-Context Identifier (INVPCID), which may reduce the performance impact on CPUs that support these features.

If you are running Windows, before updating the operating system you need to pay special attention to your antivirus systems.

To help prevent stop errors that are caused by incompatible antivirus applications, Microsoft is only offering the Windows security updates that were released on January 3, 2018, to devices that are running antivirus software that is from partners who have confirmed that their software is compatible with the January 2018 Windows operating system security update. Important: Windows security updates released January 3, 2018, and antivirus software

You need to contact your antivirus vendor to make sure you run a compatible version.

Moving down again to the operating system, you need to apply patches released by your OS vendor. Most important (if not all) OS vendors already released OS patched for supported versions.

With this, we complete our journey to patch vCenter/ESXi infrastructure against Meltdown and Spectre vulnerabilities. So far.

Happy testing!