Operating Cloud Foundry Across Multiple Data Centers with a Single BOSH Installation

by Victoria FedzkovichJanuary 25, 2016

Learn how to manage the Cloud Foundry life cycle across multiple data centers eliminating the necessity of having separate BOSH installations in each of them.

Why read this?

FOR AN ARCHITECT

Companies choose multi-data center environments for a number of reasons, including the need to optimize fault tolerance, bring applications and data closer to customers, and follow security policies. To manage Cloud Foundry workloads in multiple data centers, their operators typically have at least one BOSH installation in each of the locations.

While sticking to the mentioned method, we have also been looking for options to operate the PaaS across multiple data centers through a single BOSH Director. Find out more about our proof of concept from the article.

Table of Contents

Rationale

BOSH is a tool chain primarily associated with Cloud Foundry orchestration although the project has a more generic nature and can be used to deploy other distributed systems. When running Cloud Foundry in multiple data centers, it is a common practice to install at least one BOSH Director in each of the locations.

Interested in the scenario with several PaaS clusters, we have been exploring opportunities for their central management through a single BOSH. The motivation behind the decision to try out such an approach is the intent to save operator time and data center resources.

Preparing BOSH inputs

Multi-data center governance via a single BOSH installation requires a number of changes in a typical Cloud Foundry deployment procedure with the tool chain.

In a proof of concept, our team began making modifications from:

BOSH Director. Prepared one BOSH Director instance to work with different CPIs at the same time.
Deployment manifest. Used the cloud-config functionality to separate IaaS-specific information from the manifest and place it in .yml files—one per each infrastructure provider.
Stemcell. Uploaded stemcells for each infrastructure provider.

In the next section, find the necessary BOSH changes for performing a multi-data center Cloud Foundry deployment.

Forming a multi-data center deployment

To support a multi-data center Cloud Foundry deployment through one Director, a number of BOSH processes have to be ready for unusual conditions. Questions we have addressed are summarized below:

Process Concern Solution

Binding a deployment/creating a deployment plan Build in a multi-cloud deployment in a binding process Add the cloud entity to the BOSH database. (In the latest version of BOSH, the problem is solved with having multiple cloud-config.)

Compiling packages

Provide access to a blobstore to all BOSH Agents

Process	Concern	Solution
Binding a deployment/creating a deployment plan	Build in a multi-cloud deployment in a binding process	Add the `cloud` entity to the BOSH database. (In the latest version of BOSH, the problem is solved with having multiple `cloud-config`.)
Compiling packages	Provide access to a blobstore to all BOSH Agents	You can: Use a separate blobstore for each cloud and synchronize them. Use separate blobstores and compile packages on each cloud. Use an external blobstore accessible for all clouds. Use a DAV blobstore and share it via a VPN.
Creating jobs/VMs	Provide access to a NATS cluster to all BOSH Agents	You can: Use the “HTTPS messaging bus” mode. A BOSH Agent is able to run a HTTPS server as a coroutine and wait until instructions on what to do are passed to its endpoint as GET/POST requests. Make some NATS servers public or share them via a VPN.

You can:

Use a separate blobstore for each cloud and synchronize them.
Use separate blobstores and compile packages on each cloud.
Use an external blobstore accessible for all clouds.
Use a DAV blobstore and share it via a VPN.

Creating jobs/VMs

Provide access to a NATS cluster to all BOSH Agents

You can:

Use the “HTTPS messaging bus” mode. A BOSH Agent is able to run a HTTPS server as a coroutine and wait until instructions on what to do are passed to its endpoint as GET/POST requests.
Make some NATS servers public or share them via a VPN.

cloud-foundry-bosh-multiple-data-centers

The scheme above shows the role of a VPN server in a multi-data center BOSH installation.

Conclusion

Among enterprises, using multiple cloud infrastructure providers has been a growing trend in past few years. The logic behind such movement is simple and clear—a single solution is not always able to address all company needs.

For Cloud Foundry, the new reality brings both challenges and opportunities. Although managing the PaaS across multiple infrastructures centrally is not an out-of-the-box BOSH feature yet, doing so would be a huge step forward. Moreover, a custom BOSH-based solution we built to orchestrate multiple data centers through a single BOSH installation demonstrated once again the extensibility and potential of this open source tool.

You can join the discussion on this matter at GitHub.