- 26 Jun 2017 » Sign all the things!
Notary + LinuxKit deep dive
Just a few months ago, we open sourced LinuxKit, a toolkit to build secure, lean, and portable Linux subsystems. One of the key features of LinuxKit lies in its componentization: each of its individual components, including its kernel and system services, are completely swappable.
As you can imagine, it is important to trust each and every one of these components so you can be sure your LinuxKit subsystem is assembled securely. In particular, you should be able to understand where the components come from, what they are supposed to include, and when you should stop using them in case they become out of date. This is a hard requirement for every package – after all, a system is only as secure as its most vulnerable component.
With this goal in mind, the LinuxKit team has been using Notary, a signing and verification framework that is also part of the Moby Project, to make this easy. Today, every LinuxKit package is signed with Notary. We’d like to tell you more about what this means for your LinuxKit subsystems today, and what you can expect in the future.
Cryptographic signing is an absolute necessity for establishing trust over content sent over an untrusted media such as the internet. While there are many cryptographic signing systems available for use, most only provide integrity and authenticity guarantees - that the package contents were not modified, and that it was signed off by a key you trust.
Notary not only provides integrity and authenticity, but goes above and beyond by providing survivable key compromise and freshness properties.
By providing survivable key compromise, Notary ensures that any key can be seamlessly rotated out without breaking any trust or functionality in the system. This is particularly important if part of your signing process occurs in online, more exposed environments, where keys are more likely to be exfiltrated. Additionally, it’s difficult in many other signing frameworks to rotate the keys without discarding the root of trust or manually configuring subkeys.
Freshness ensures that you cannot download an old package, or get frozen in time on a stale package. This is a common issue in many signing systems: because it is difficult to invalidate or expire signatures, old and potentially vulnerable packages often carry still-valid signatures (ex: openssl with Heartbleed).
Notary achieves these guarantees by signing separate metadata to capture context about its packages. It’s worth mentioning that Notary and the signed metadata it produces is an implementation of The Update Framework, which is a research project based from NYU that has been published and discussed in several peer-reviewed academic publications.
Notary + LinuxKit: better together
Perfect for our packaging
LinuxKit uses Docker images as the distribution format for all of its packages, including its kernel and init process, and the Docker Hub to store them. While Notary can sign over any type of content, Docker includes a robust and easy-to-use integration known as Docker Content Trust. By simply exporting
DOCKER_CONTENT_TRUST=1, Docker will verify all signatures on pull, and sign images on push.
We also implemented a trust validation step in the
mobybuild tool by leveraging the Notary library. Any packages that are specified by image or organization name in the LinuxKit yaml file will be verified using the Notary library. Today, all LinuxKit packages in the
linuxkitorganization in Docker Hub are signed:
Collaboration made easy
Each LinuxKit maintainer has their own key for signing purposes. These keys map to less-privileged roles in Notary known as delegations - not only does this prevent the need for sharing any private key material, but it also works seamlessly with Docker Content Trust, such that
DOCKER_CONTENT_TRUST=1 docker push linuxkit/kernel:4.9.xpushes and signs with the appropriate keys. That said, the LinuxKit team went an extra step further to ensure that every Makefile for LinuxKit packages includes Docker Content Trust on every pull and push, so running
makealso works for convenience.
As we can see above, each maintainer has their own key and role, and there are a couple of extra roles for release management and CI. We can easily determine which role and key signed off on a package because that context is captured in Notary’s metadata, for example Rolf has signed off on several of the 4.10 series kernel packages:
Using Notary delegations, we can not only audit who signed the packages, but enforce policy about the signatures on a package. For example: a package could require a simple threshold of two signatures, or even signoff from two maintainers and CI to be considered valid.
A trusted package manager
Today, Notary and Docker Content Trust are deeply integrated into all LinuxKit packages, not only providing strong cryptographic and usability features, but ultimately equipping LinuxKit with a trusted package manager.
A great example of this integration is how the
mobytool leverages Notary during offline builds for a LinuxKit image. If the
mobytool detects it is offline when trying to update to the latest signed data for an individual package, it will fall back to already trusted data from a previous validation if the data is still unexpired. It will perform the same signature verification step using the key data it had previously trusted, and output a warning to the prompt to inform the user that there may be a more up to date version remotely for when they can get back online:
Overall, the LinuxKit team is excited to leverage Notary’s state-of-the-art signing and verification features, and we view it as a fundamental step to establish trust in our packages - to us, if a LinuxKit package isn’t signed by Notary, it should be as if it doesn’t exist.
Going forward, there are many features of Notary yet to explore: fine-grained auditing, signature threshold policy enforcement, and even key pinning to our roots of trust. We look forward to more advanced integrations between Notary and LinuxKit.
- 26 Jun 2017 » Moby Summit June 2017 Recap
Moby Summit at Docker headquarter
On June 19 2017, 90 members of the Moby community gathered at Docker headquarter in San Francisco for the second Moby Summit. This was an opportunity for the community to discuss the progress and future of the Moby project, two months after it was announced.
We started the day with an introduction by Solomon Hykes, and a look at the website redesign: the Moby project website now has a blog, an event calendar, a list of projects, and a community page with links to various community resources. The website code is open source, issues and PRs to write guest blog posts or enhance it are welcome.
Then each team gave an update on their progress: Linuxkit, containerd, InfraKit, SwarmKit and LibNetwork, as well as the three new Moby Special Interest Groups, Linuxkit Security, Security Scanning & Notary and Orchestration Security. All these talks have been recorded and you can find the videos and slides below.
In the afternoon, we split into 5 Birds Of Feathers (BOF) sessions: runc/containerd, LinuxKit, InfraKit, Security, and Security Scanning. You can find links to the BOF Notes at the end of this post.
We ended the day with a recap of the BOF sessions, and some beer.
Moby Summit Introduction
Slides: Moby Summit Introduction
Slides: LinuxKit update
Slides: Infrakit update
Swarm Proxy is a program for managing Docker Swarm services behind a reverse proxy. It uses the Docker events stream to monitor for services being created and removed, and then uses Swarm Configs to update a reverse proxy service to correctly route traffic to those services. It is stateless.
Slides: Libnetwork update
Finally, the Docker security team gave the update on the following topics:
- LinuxKit security update
- Security scanning and Notary update
- Orchestration security update
Slides: Orchestration security
BOF Summary Notes
- 11 May 2017 » Moby Summit at DockerCon 2017-05-11
Moby Summit videos from DockerCon
Last month at DockerCon, Solomon introduced the Moby Project: a new open-source project to advance the software containerization movement. The idea behind the project is to help the ecosystem take containers mainstream by providing a library of components, a framework for assembling them into custom container-based systems and a place for all container enthusiasts to experiment and exchange ideas.
- 08 May 2017 » Moby Dev Report 2017-05-08
- 08 May 2017 » Moby Builder Report 2017-05-08
Quality: Dependency interface switch
- 01 May 2017 » Moby Dev Report 2017-05-01
This is the 1st report, since the Moby project was announced at DockerCon. Thank you to everyone that stayed an extra day to attend the summit on Thursday.
- 01 May 2017 » Moby Builder Report 2017-05-01