Cloud Foundry’s Garden: Back Ends, Container Security, and Debugging

Containerization in Cloud Foundry

Providing isolation for applications, containers have always been a core component of Cloud Foundry—each application deployed to the platform runs inside its own container.

So far, two mechanisms have been developed for creating and managing Cloud Foundry containers: Warden and Garden. The Droplet Execution Agent (DEA), the first Cloud Foundry’s runtime architecture, relied on Warden to orchestrate application containers. After the DEA was rewritten, a newer Diego runtime came along, together with Garden containers. (At the recent CAB call, it was highlighted that Diego 1.0 is officially available and capable of managing 250,000 containers.)

In this article, we provide a brief overview of how the Cloud Foundry containerization layer has changed over time. For more technical details, check out our previous post on Garden internals.

How Garden is different

In Cloud Foundry versions that use Diego, Garden replaced Warden.

Warden—the older container technology—was mostly written in Ruby, with some C code. It leveraged several features of the Linux kernel—such as, namespaces and control groups—to provide process isolation and resource management for containers.

Warden clients (DEAs) talk to Warden via a protocol that uses Protocol Buffers. The original container technology has a monolithic architecture: the Warden server is coupled into a single application with the container manager responsible for container life cycle routines. Such a design, however, makes it difficult to plug in an alternative container implementation.

Unlike Warden, Garden was implemented in the Go programming language, and it has a modular architecture. To ensure flexibility of the system, the Garden server was decoupled from the container manager. The manager formed a separate component—back end.

While the server and client are platform independent, back ends enable support for specific platforms (Linux and Windows). Today, Garden containers are used in several products, including Cloud Foundry itself, Concourse CI, and BOSH Lite.

Garden back ends

For creating and managing containers, Garden provides a platform-neutral Go API and multiple back ends, with each back end implementing the Garden API for a specific platform. By now, three Garden back ends have been created:

• Garden-Linux (phased out)
• Garden-runC (Guardian)
• Garden-Windows (Greenhouse)

Garden-Linux

The retired Garden-Linux back end was the successor of the Warden container implementation with similar features for providing process isolation, including the mentioned Linux namespaces, control groups, and a layered file system.

In addition to the default buildpack life cycle, Garden-Linux introduced support for Docker containers in Cloud Foundry.

The project, however, is no longer actively developed and has been moved to Cloud Foundry Attic. The reasons behind switching to Garden-runC are explained in this article from the Garden team.

Garden-runC

Garden-runC (Guardian) is a Linux back end for Garden that uses runC to spawn and run containers according to the Open Container Initiative (OCI) specification. In Cloud Foundry, Guardian replaced Garden-Linux and now serves as the default back end for Garden containers.

“In 2015, Open Container Initiative appeared to create industry standards around containers to build an open, portable, platform-, cloud-, and hardware-independent container and runtime format.” —Maksim Zhylinski

Concerned with developing open industry standards around the container image format and runtime, OCI currently provides two specifications:

• Image specification defines how container images should be created and prepared to run.
• Runtime specification defines how to run the container images.

Garden-Windows

Garden-Windows (Greenhouse) is a Windows back end for Garden. Since Windows does not support containers in the same way as Linux does, a different approach was needed for developing this back end.

To provide support for Windows containers, the following features are used:

• File system isolation: creating a unique user per container
• Disk usage limiting: setting NTFS quotas
• CPU and memory usage limiting: employing Windows job objects
• Network isolation: binding applications directly to the external IP of the cell

Security considerations for Garden

Garden uses the core features of the Linux kernel to ensure the security of Linux-based containers in Cloud Foundry. The namespaces and cgroups have been around for years and become the cornerstone of many popular projects.

In the case of the Greenhouse back end, it is important to understand that a completely isolated file system cannot be created for Windows containers. Parts of the host file system are still visible to the temporary user who owns the container. It is better to avoid storing sensitive information in locations where the containers can access it, such as C:\Program Files.

At the same time, some principles of the Garden architecture that promote container security might put certain constraints on the types of operations you can perform.

For instance, every container in Garden uses its own subnet with interfaces, and a firewall is created for the container. Although it is good from the perspective of security, these rules make connecting containers with each other an issue.

“It doesn’t really fit into the PaaS ideology. For example, in Docker, you have a single subnet for your containers, and you can create links between them. In Garden, you can’t.” —Maksim Zhylinski

Debugging in Garden

It is possible that your application running in a Garden container can one day fail to work as expected. Logs are not always sufficient to identify the reason, so you might want to get inside the container and check what is happening there.

Diego and Garden simplify application debugging by providing a CLI command—cf ssh APP_NAME—that brings you directly inside the container (image below).

Before this command was released, getting into an application container had been way more complicated and included multiple steps. The user also needed to have access to BOSH.

If you are interested in debugging the container manager itself, there is a tool, gaol, which is a CLI for Garden. You can create containers, delete them, open a shell inside a new container, and do many more things with it.

To sum up, the newest Cloud Foundry container technology—Garden—provides a platform-independent API for creating and managing containers. Pluggable back ends, which are the platform-specific implementations of the Garden interfaces, ensure that users can securely run applications in both Linux and Windows-based containers.

When reading this, keep in mind that Diego and Garden are being actively developed, so things are changing pretty fast.

Related reading

• A New Cloud Foundry Container Engine for Linux, Garden-runC v1.0, Is Out!
• Cloud Foundry Containers: The Difference Between Warden, Docker, and Garden
• Cloud Foundry Security: Do Containers Contain?

Want details? Watch the video!

Related slides

About the speaker

Maksim Zhylinski is a Cloud Foundry Engineer at Altoros. He is an expert in cloud computing, networking, and Cloud Foundry, having worked on multiple BOSH CPIs, releases, and service brokers. Maksim has 6+ years of experience in Ruby, JavaScript, and Go, as well as extensive skills in server and client-side web app development. He is an active member of the Ruby and Go communities and a frequent contributor to various open-source projects.