From globally redundant datacenters to SMB server rooms, if it’s using PXE booting with IPv6 enabled, the admins probably have emergency patching to do.
PixieFail
Quarkslab published a blog post on PixeFail, a set of 9 vulnerabilities in a popular firmware component. Think “Windows is a popular user OS” levels of popular applied to firmware needs in the world of large hardware & cloud providers. TianoCore is a “community supporting an open source implementation” of UEFI, including a development environment named EDK II (commonly alternatively spelled “EDK2”). This environment is used by several, if not a vast majority of, large vendors for their firmware needs, and the PixieFail vulnerabilities are in the IPv6 network stack of EDK2’s PXE component.
Firmware allowing for malicious activity unbeknownst to the host OS is a known security challenge, especially in the enterprise. Ars Technica has nice summary of valuable info in their article on PixieFail. See the snippet below, emphasis mine.
The implementation is incorporated into offerings from Arm Ltd., Insyde, AMI, Phoenix Technologies, and Microsoft.
By exploiting the PixieFail vulnerabilities, an attacker can cause servers to download a malicious firmware image rather than the intended one. The malicious image in this scenario will establish a permanent beachhead on the device that’s installed prior to the loading of the OS and any security software that would normally flag infections.
Attackers need not establish their own malicious server or gain high-level privileges. Instead, the attacker only needs the ability to view and capture traffic as it traverses the local network.
Dan Goodin, Ars Technica
Public Disclosure
The blog post by Quarkslab is a highly technical read beyond my expertise, albeit with well-written summaries in the intro and conclusion sections, but of particular interest to me is some select content in the disclosure timeline.
Your typical responsible disclosure of new security vulnerabilities starts with the research team notifying the affected providers of the discovered vulnerabilities with notes on how to reproduce exploitation. Public disclosure is normally performed 90 days after providers are informed. This timeframe is the industry standard meant to strike a balance. Providers need time to patch, but every day the public is not informed, malicious actors are one day closer to discovering and exploiting the same vulnerabilities independently of the good guys (if they haven’t been doing so already).
The 90-day deadline also gives affected providers extra incentive to patch their systems in a timely manner rather than leaving a potentially large threat at low priority, or worse, ignoring the threat altogether. The standard is not at all iron-clad and may be changed depending on the prudent judgement of all involved parties (researchers, providers, and sometimes government entities), but 90 days is certainly the most common.
Quarkslab’s Disclosure Timeline
What makes some of the provider communications in this case so interesting is that not only did multiple affected providers repeatedly ask Quarkslab to delay public disclosure or limit technical exploit content in their disclosure, but delay disclosure from the original date of November 2 to the “middle of 2024,” approximately 10 months from the August 8 date of initial report and far beyond the standard 90 days Quarkslab initially planned for.
I know enough to know when I’m out of my depth, and this is such a case in both technical and “political” aspects. My only insight is generalized: Large vendors have incredible and complex technical infrastructure and are often subject to the same limitations as the rest of us in time, expertise (TianoCore noted one of its leads retired), and producing value for stakeholders in sometimes nuanced ways. It’s also important for researchers to be able to disclose the results of their efforts to produce value for their own organizations and provide benefit to the public if possible.
With that said, I’ve included some selections of the timeline entries below that were of interest to the relative layman I am. Note that each item but the last is shown in full. The last item has been shortened for brevity, and all emphasis is mine.
- 2023-08-18 AMI informed that Tianocore’s PSIRT lead had retired and explained that once a new one was assigned, a volunteer would pick up the issue. Indicated that November 2nd, 2023 was too optimistic for a deadline and asked if Quarkslab had plans for public disclosure.
- 2023-09-06 Insyde Software asked Quarkslab if the publication date could be postponed by 30 days, as the issues had to be addressed by 4 sets of PSIRT teams (Tianocore, IFVs, ODMs, OEMs).
- 2023-09-06 Quarkslab agreed to re-schedule disclosure to December 1st, 2023 and said that so far only Insyde Software indicated it was affected and addressing the vulnerabilities.
- 2023-10-23 Microsoft asked if it was possible to postpose [sic] the Dec. 1, 2023 disclosure date as they would need more time to deploy a complete fix on their cloud infrastructure and depended on some partners for it. Also asked if Quarkslab was to publish the details in a blog post.
- 2023-10-23 Quarkslab agreed to reschedule disclosure to December 7th, 2023 but reminded that, as stated in August, needed to publish the research work within 2023. It also indicated that since vendors were already progressing towards releasing fixes, it did not think reasonable to delay disclosure much longer, and confirmed that details about the vulnerabilities would be published in a blog post.
- 2023-10-25 Microsoft said that postponing disclosure one week was immaterial as it would not allow sufficient time to roll out comprehensive fixes to customers. Asked for the specific technical content of the blog post to be published, expressing concerns that exploit code would be included and asked if it was possible not to do so. Indicated that they had confirmed with Tianocore that patches were not finalized yet and were unlikely to be available before the December disclosure date and that they disagreed with Quarkslab’s assumption that “vendors were progressing towards releasing fixes”. Microsoft asked to hold off disclosure until May 2024.
- 2023-10-25 A Tianocore core developer requested more time to get products patched and clarified that there were no validated patches on the open source side yet, a first draft had been submitted for review and the expectation was they’d become available in November to Tianocore’s infosec community. Patches would then have to be integrated, tested and deployed to customer devices which usually takes months. The core developer referenced a UEFI Forum paper which describes the complex supply chain challenges of the UEFI ecosystem, and indicated that “any public announcement before middle of 2024 would cause significant negative impact.”
- 2023-10-25 Phoenix Technologies expressed strong support for Tianocore’s and Microsoft’s position and indicated that the ongoing case was mentioned in a podcast (without providing any technical details) to exemplify the difficulties of producing patches and why embargo lengths must be flexible.
- 2023-11-11 Microsoft asked Quarkslab to share a technical draft of the contents that would be published in the blog, reiterated its concerns that it would contain explicit details regarding exploit code and asked for the exploit code not to be added.
- 2023-11-13 Dell agreed with Microsoft’s position that delaying disclosure one week did not make a difference, and said that a later disclosure (May 2024) would be more suitable.
- 2023-11-14 Summarized its position with the following: Quarkslab was being asked to extend the embargo period from 120 days (which had already been extended from the original 90) to about 10 months counting from initial report date, without any firm commitment from any vendor to an actual release date and an opaque timeline.
From that point the timeline details Quarkslab contacting CERT-FR for them to weigh in on a public disclosure date, parties agreeing to wait to publish until 2024 due to code freezes, and eventually publishing on January 16.
Honestly, I’m not sure what I expected to get out of the timeline, but it wasn’t that.