Mythbusting the scope of Livepatch protection

The purpose of this article is to share the technical realities of security patching for the Linux kernel, and the intended scope of the Linux kernel’s livepatch capability. We’ll cover when kernel live patching is most appropriate, and when updating deb and snap packages and then rebooting is the best option. 

Canonical Livepatch is a service that allows Ubuntu long-term support (LTS) and Ubuntu Core users to apply critical kernel security patches without rebooting. Canonical Livepatch delivers live, rebootless security updates for high-priority kernel vulnerabilities on Ubuntu LTS and Ubuntu Core systems, and is included with Ubuntu Pro. Ubuntu Pro is a subscription for security, hardening, compliance, and support for open source software.

In this article, we’ll run through Livepatch in more detail, in order to bust the myths that often arise around the subject of kernel live patching. We’ll highlight when live patching is the most effective option, and run through instances where a reboot is the more appropriate option. Crucially, we’ll explain why live patching complements updating deb packages and then rebooting, and why live patching isn’t a full-on replacement for patching all CVEs in all situations.

Canonical Livepatch is built for Ubuntu LTS and Ubuntu Core, and is tightly coupled with the application binary interface (ABI), the layer between user-space applications and the operating system’s kernel. Extensible Firmware Interface (EFI) Secure Boot and Linux Kernel Lockdown are complementary security features that work together to protect the boot process and the running kernel, and Canonical Livepatch conforms with these standardized security processes and their chain of trust requirements. Canonical Livepatch does not insert updates through proprietary or nonstandard mechanisms that bypass these kernel lockdown rules, and guarantees the updates only come from a trusted publisher. Enterprises using Ubuntu choose Canonical Livepatch for this precise integration.

Contact us to get started

Myth #1: every vulnerability should be live patched

A common misunderstanding we encounter is that live patching is desirable for every CVE, in every situation. However, not every CVE requires urgent attention or can be safely live patched without increasing operational risk. Live patching the kernel in memory is an exploit window mitigation solution, and is not a comparable substitute for installing software updates with apt, snap, or Landscape. Indeed, as others suggest, live patching 90-95% of vulnerabilities would be impractical, and doing this could introduce operational risks that hang machines. It’s likely that users are left looking for rollback capabilities, when this happens.

Canonical Livepatch patches kernel vulnerabilities with critical and high Common Vulnerability Scoring System (CVSS) and Ubuntu Priority ratings. Canonical targets vulnerabilities that have security implications, such as privilege escalation or remote code execution, and are practical to patch safely in-memory. 

Additionally, Canonical Livepatch does not patch userspace libraries like OpenSSL or glibc, because that is the responsibility of unattended-upgrades or a systems management tool, like Canonical Landscape.

Myth #2: Live patches can trap you in a patched state

There are two primary approaches to kernel live patching: incremental and cumulative patching. Incremental live patching stacks one security update upon another, and patching live-running code with in-memory overrides on an incremental basis can become fragile over long periods, especially when these updates are stacked in perpetuity without prior testing on a variety of hardware configurations. In this scenario, it’s understandable why readers believe in the need for rollback functionality, in order to resolve any unexpected interactions between patches, which can cause instability.

However, Canonical Livepatch eliminates the risk of instability by providing cumulative patches instead of stacking incremental ones, which makes rollbacks unnecessary in practice. 

Before being published, Canonical runs every livepatch through the same testing and rigor as their kernel packages. Each patch is tested cumulatively with previous patches on real hardware, without emulation. Livepatch Client downloads a single cumulative patch for the running kernel. In practice, the failure rate of Canonical Livepatch deployments is vanishingly small, and for Canonical Livepatch, reboots remain the safest and most sensible rollback path.

Myth #3: Live kernel patching is always preferable to installing updates and then rebooting

As mentioned in our introduction, you should see live patching and installing an updated kernel package followed by rebooting as two security patching methods which complement each other – not as two alternate approaches to security. Canonical Livepatch provides a timely, quick solution to critical CVEs, as these need to be actioned immediately. By patching in the live kernel, Canonical Livepatch shrinks the exploit window for a given CVE and keeps your customers safe, without requiring a reboot associated with deb or snap package updates for the kernel. 

However, Canonical Livepatch is not a replacement for updating the kernel package and then rebooting. It is a tool that gives you more control by preventing a required unscheduled deb or snap kernel package update and reboot. Indeed, updating the kernel deb or snap package and then rebooting on a schedule is the responsible approach for security hygiene. No enterprise Linux operating system publisher recommends avoiding reboots indefinitely. Kernel modules like NVIDIA drivers often can’t be reloaded safely without updating deb packages and rebooting. In some edge cases, kernel updates can’t be livepatched due to ABI changes or complex restructuring. Security vulnerabilities exist in userland packages, and installing security patches and restarting services is not sufficient (or applicable) in certain situations:

• libc6 is fundamental to virtually every process on a Linux system, and is packaged as glibc on Ubuntu. Even small security patches require restarting all linked processes, or more practically, a reboot. When updating libc6 even if services are restarted manually, some processes such as systemd, won’t reload libc6 cleanly. True safety comes with rebooting after updating standard libraries.
• Intel and AMD microcode in the cpu-microcode package is loaded at boot or via a special mechanism at runtime. Runtime updates apply to cores already idle, or not running privileged code. To fully ensure all cores and threads operate with the updated microcode, updating the deb or snap package and then rebooting is required. The Spectre and Meltdown vulnerabilities were remediated with cpu-microcode updates.
• The systemd and init packages are critical to system management, if vulnerabilities exist here. Mitigations for systemd require a restart of PID 1, which is impractical without rebooting.
• dbus is a package responsible for inter-process communication (IPC) that allows applications to communicate with each other, and updating the deb package and then rebooting ensures a clean state.
• udev is a package which monitors kernel events and applies rules to create, configure, and control device nodes in the /dev directory. Changes to udev (by way of installing a software update) require a reboot to ensure consistency.
• Patches to crypto libraries such as openssl, gnutls, and libssl require restarting all dependent processes, after updating deb packages. Technically no system reboot is required, but practically it’s safer and faster to reboot after updating libraries, than to manually hunt down and restart everything safely.
• Updates to hypervisors require restarting running virtual machines. Rebooting hypervisor hosts is often preferred, operationally.
• Patches to display servers such as xorg and wayland require restarting the session, potentially disrupting desktop environments. While this is not a full reboot, it is disruptive enough that in practice it might as well be.

Reboots cleanly flush accumulated state inconsistencies from memory leaks, hung file handles, and other problems that could fester with long uptime. Eliminating reboots would expose any operating system to these inefficiencies over time.

In short, trying to avoid all reboots results in incomplete security patching coverage, and leads to fragile patching strategies that involve service restarts and error prone dependency tracing. Rebooting provides a guaranteed clean slate.

Conclusion

Kernel live patching is part of a broader security strategy. Kernel live patching is not a replacement for traditional patch management, which includes updating deb packages and then rebooting, and defense-in-depth – however, tools like Canonical Livepatch provide you with more control over when, and how, reboots occur for the most critical vulnerabilities.

To ensure the highest levels of security, stability, and integrity in your Linux servers, performing periodic routine maintenance that includes deb package updates and a system reboot is not just recommended, it is essential. Annual, bi-annual, and more frequent security patching strategies and their respective pros and cons are covered in the “Best Practices for Scheduling Security Patching Automation” webinar.

Canonical Livepatch is provided as part of Ubuntu Pro, a subscription which also includes other security patching solutions that cover over 30,000 open source software titles and their 100,000+ package dependencies, for up to 12 years. Organizations value Canonical’s transparency around testing and process, and trust Canonical’s machinery to build their kernels. That’s why organizations trust Canonical to use that same testing, process, and machinery to publish kernel livepatches. As the publisher and maintainer of Ubuntu, Canonical’s Ubuntu Pro bundle which includes Landscape and Livepatch is a smart fit for any organization.

To find out more about Canonical Livepatch, what customers have to say about it, and how you can enable it in your organization, visit our Livepatch page.

Further reading

• The Canonical Livepatch datasheet
• Blog: An overview of live kernel patching 
• Webinar: Best practices for scheduling security patching automation
• Webpage: Ubuntu Pro
• Landscape