Maturity of App Deployments on Kubernetes: From Manual to Automated Ops

Different stages of maturity

Kubernetes is ubiquitous in the ability to run on any platform. This way, organizations can target customers and shift their workloads wherever they need to be run. However, Kubernetes itself is a massive platform, so businesses looking to adopt the technology will likely face some challenges associated with:

• container ad platform management
• complexities of configuration
• stateful apps requiring additional architectural solutions
• the necessity to strike balance between containers and managed services

To alleviate these challenges, companies can make use of continuous integration and continuous delivery (CI/CD). Implementations of CI/CD pipelines may vary from case to case, but there are some common goals:

• accelerating the delivery of new, high-quality services
• simplifying environment management
• reducing the impact of code changes
• automating operations
• gaining insights across resources and apps
• protecting customers and the business

Barry Williams

Depending on the maturity of a Kubernetes deployment, there exist scenarios that favor either manual or automated operations. Barry Williams and Michael Jacobi of Altoros, as well as Curtis Rissi of AWS, discussed four stages of maturity in Kubernetes deployments during a recent webinar. The speakers also explained how the implementation of CI/CD in each stage affects application deployment from manual to fully automated.

“Kubernetes is all about velocity and being able to deliver software very quickly. In order to keep up with that velocity, you need to have automation in place that can thoroughly test everything you are deploying, because you do not have time for manual testing.” —Barry Williams, Altoros

The beginner stage

In this stage, manual deployments are common as applications are still quite simple. It is also not unusual for organizations to have a monolithic architecture, which can later serve as a starting point when making the shift towards microservices.

While deployments are still done manually at this stage of Kubernetes maturity, there are still some benefits:

• The barrier to entry is low.
• Organizations do not need complex infrastructure.
• Manual deployments are faster for one-off processes.
• The support team does not need to learn additional products.
• The beginner stage is good for learning Kubernetes and can serve as an entry point for automation.

Manual deployments in the beginner stage (Image credit)

“At this point, developers, operators, and QA are still different teams. The developers make a piece of code and throw it over the wall. The operators will pick up on that, manually edit some YAML files, and then throw that over the wall. QA then does their testing. The problem here is the turnaround time.” —Barry Williams, Altoros

Due to the nature of manual deployments, there are also challenges:

• Manual updates and rollbacks increase the risk of configuration mistakes.
• Management is harder and requires human intervention.
• Hardcoded values in configuration files require significant effort to update.
• There is a potential lack of standardization.
• Rollbacks and repairs take longer.
• Microservices are more complex to maintain.

Despite the lack of automation, manual deployments are still viable in certain scenarios:

Michael Jacobi

• working on proof of concepts
• running environments with infrequent updates or simple architecture
• having no strict audit and compliance requirements

“Manual deployments are human-driven. Developers are manually building their applications and pushing container images into a registry, operators are handcrafting YAML files, and QA are testing manually. It is certainly suitable for some scenarios, especially when you are just starting.” —Michael Jacobi, Altoros

The intermediate stage

Once organizations are in the intermediate stage, there are still manual processes, but some level of automation is introduced to help perform more frequent updates. The addition of automation creates a deployment standard, which helps if there are compliance requirements.

The benefits of basic automated deployments at this stage include:

• Clusters are easier to create.
• Test automation and metrics can be embedded in the deployment cycle.
• Rollbacks and repairs are faster.
• Basic automated deployments can generally be run by Tier 1 support teams.
• Deployments can be integrated with project-tracking products.

Basic automated deployments in the intermediate stage (Image credit)

“Developers have a CI process to ensure that your container images are built the same way every time. Operators may still be manually crafting some YAML files, but you may also have a process where CD is beginning to grow. You can have an automated process for applying YAML changes. What I usually see is that the CD process is actually the CI just automating kubectl apply, so it is still pretty basic. Then, you have QA come in for testing.” —Barry Williams, Altoros

The intermediate stage of Kubernetes maturity has its own challenges:

• A skilled team is required to integrate CD with existing systems.
• Rollbacks are still partially manual without proper testing and monitoring.
• Additional resources are required to maintain the CD solution.
• Secrets need to be injected into the pipeline.

Adding basic automated deployments make sense in the following scenarios:

Curtis Rissi

• running environments with frequent deployments
• operating Agile teams
• having audit and compliance requirements
• deploying complex applications
• running multiple environments (development, stage, and production)

“The key thing here is that you are taking the low-hanging fruit, the easiest portions to automate, and starting automation. You are giving yourself a foundation that you can then iterate on and build some of the more complicated things.” —Curtis Rissi, AWS

The advanced stage

The advanced stage of Kubernetes maturity introduces blue-green deployments with Canary and automated testing. According to Curtis, the idea behind Canary is to test with an actual production server. A traditional blue-green deployment is a straight cutover, where you spin up two environments. One is running the current version, the other the new version. With Canary in that mix, you still have both environments spun up, but you’re only testing a small portion and you ease the cutover.

The benefits of largely automated deployments at this stage include:

• Organizations can deploy anytime with predictable or no downtime.
• Rollbacks are faster and controlled.
• Customer experience is better.

Largely automated deployments in the advanced stage (Image credit)

“In the intermediate level, you’re probably using your CI tool for CD. At this level, you have to get a specialized tool for CD. Additionally, you are going to be moving towards a DevOps model. Developers, operators, and QA engineers start to become the same person or at least the same team, where people are working together to perform different functions, but they’re doing it highly coupled.” —Barry Williams, Altoros

The challenge in this stage of maturity is getting DevOps used to the combination of blue-green deployment and Canary. Once achieved, DevOps engineers have more time to focus on creating new features.

Scenarios where largely automated deployments come useful are:

• running under strict service-level agreements
• running very large deployments
• having limited deployment personnel

“The key here is to not treat it as a separate production environment, but more like production plus. In order to get the signal from the noise and get that one instance out of all the monitoring and logging from the rest, you have to be able to single it out. Otherwise, an error might be buried under the rest of that production environment. You want to treat that as the test, and if everything goes well, expand to another.” —Curtis Rissi, AWS

The expert stage

The final stage of maturity involves full automation and advanced routing. The expert stage also introduces the concept of GitOps, which according to Curtis, means that everything is in a declarative state in versioned artifacts, so the environment itself describes how it should run.

Service mesh is also introduced at this stage. It provides advanced routing through more granular control over services. Service mesh also enables policy enforcement between services.

Overall, fully automated deployments provide several of the following benefits:

• declarative, versioned artifacts
• granular control over a data flow
• weighted traffic distribution
• request limits
• circuit breakers
• enhanced security
• mutual TLS
• access control
• enhanced observability
• distributing tracing
• service metrics
• fault injection testing

Fully automated deployments in the expert stage (Image credit)

“The main difference in this stage is that you will be using a GitOps tool. Inside Kubernetes, we now have a GitOps Operator that listens to a Git repository. Developers, operators, and QA engineers push changes to this Git repository, and the GitOps Operator takes that information and makes it a reality.” —Barry Williams, Altoros

Naturally, the addition of service mesh itself is the challenge as it adds complexity to the overall architecture and may have some performance implications. That said, service mesh is ideal when running such scenarios as:

• maximizing cost and time efficiencies
• deploying complex microservices-based apps with dependencies
• running advanced security protocols

AWS deployment tools

In order to help with organizations get started with deployment automation, Curtis shared a few tools and services from AWS.

• CodeCommit is a fully managed source control service that hosts secure Git-based repositories.
• CodeBuild is a CI service that compiles source code, runs tests, and produces software.
• CodePipeline is a CD service for automating release pipelines.
• CodeDeploy is a service that automates software deployments to various compute services.
• X-Ray is an introspection tool that helps developers analyze and debug applications.
• CloudWatch is a logging and monitoring tool.

Software release steps with AWS tooling (Image credit)

Regardless of which tools or services an organization use, being able to fully automate a deployment through CI/CD should be the goal. Businesses that follow this practice are able to ship faster and develop more features.

“As people transition to more automated deployments, they will see 5x less failure rates, because there are less manual operations. The rate at which you can get change to your customers goes up to 440x faster from commit to deploy. If things are going more safely and quickly, you can make 46x more frequent deployments. You also spend less time on undifferentiated heavy lifting and 44% more time on feature sets.” —Curtis Rissi, AWS

Automating makes deploying to production trivial. Since everything is delivered quickly, failure is no longer something to fear and becomes a natural part of the process.

Want details? Watch the webinar!

Related slides

Further reading

• Automating Event-Based Continuous Delivery on Kubernetes with keptn
• Automating Deployment of Pivotal Container Service on AWS
• Rule-Driven Automation on Kubernetes with Autopilot Monitoring

About the experts

Curtis Rissi is Senior Solutions Architect at AWS. Over the past 15 years, he has worked at multiple levels of IT, including development, infrastructure, architecture, and management. Curtis is highly skilled at seeing connections between technologies and putting them together in ways that solve very complex problems. He prides himself on his ability to develop the long-term vision and creating simple, intuitive solutions that address real business needs.

Barry Williams is Cloud Solutions Architect at Altoros. He has designed and implemented the architecture for microservice applications. Barry focuses on automation in all things and cloud platforms. He is an expert in Kubernetes, Cloud Foundry, Docker, Linux, Concourse CI, ELK Stack, Shell Script, Python, Java, Go, Ruby, Agile, Extreme Programming, etc. Barry is also a Google Cloud Certified Professional Cloud Architect.

Michael Jacobi is Director of Global Field Services at Altoros. He manages DevOps, SRE, and Developer consultants globally. Michael also defines standards and develops product offerings. He provides high-value systems integration consulting and training services, focusing on building massively scalable cloud-native systems with Infrastructure as Code and GitOps.