{"id":60779,"date":"2021-03-31T18:42:25","date_gmt":"2021-03-31T15:42:25","guid":{"rendered":"https:\/\/www.altoros.com\/blog\/?p=60779"},"modified":"2022-03-21T13:57:28","modified_gmt":"2022-03-21T10:57:28","slug":"lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment","status":"publish","type":"post","link":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/","title":{"rendered":"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79_2 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#Scaling_too_fast\" >Scaling too fast<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#Creating_a_hybrid_environment\" >Creating a hybrid environment<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#Migration_roadmap_and_results\" >Migration roadmap and results<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#Managing_multiple_clusters\" >Managing multiple clusters<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#Want_details_Watch_the_videos\" >Want details? Watch the videos!<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#Further_reading\" >Further reading<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#About_the_experts\" >About the experts<\/a><\/li><\/ul><\/nav><\/div>\n<h3><span class=\"ez-toc-section\" id=\"Scaling_too_fast\"><\/span>Scaling too fast<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Lyft is one of the largest ridesharing companies in the US, operating in all 50 states, as well as in Toronto, Ottawa, and Vancouver. Headquartered in San Francisco, the company reported a total revenue of $2.3 billion in 2020.<\/p>\n<p>A few years after its founding in 2012, Lyft had already outgrown most of its infrastructure. &#8220;The company grew very quickly,&#8221; explained <a href=\"https:\/\/www.linkedin.com\/in\/vickicheung\/\" rel=\"noopener noreferrer\" target=\"_blank\">Vicki Cheung<\/a>, Engineering Manager at Lyft, during an <a href=\"https:\/\/softwareengineeringdaily.com\/2020\/01\/23\/lyft-kubernetes-with-vicki-cheung\/\" rel=\"noopener noreferrer\" target=\"_blank\">interview<\/a> with Software Engineering Daily. &#8220;They were outgrowing both in scale and in the number of engineers that were using the system.&#8221;<\/p>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-2015-Infrastructure.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-2015-Infrastructure-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60968\" \/><\/a><small>Lyft&#8217;s monolithic architecture back in 2015 (<a href=\"https:\/\/www.slideshare.net\/datawire\/lyfts-envoy-from-monolith-to-service-mesh-matt-klein-lyft\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<div id=\"attachment_60889\" style=\"width: 110px\" class=\"wp-caption alignright\"><img decoding=\"async\" aria-describedby=\"caption-attachment-60889\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Vicki-Cheung-150x150.jpg\" alt=\"\" width=\"100\" class=\"size-thumbnail wp-image-60889\" \/><p id=\"caption-attachment-60889\" class=\"wp-caption-text\"><small>Vicki Cheung<\/small><\/p><\/div>\n<blockquote><p>&#8220;Like a typical startup story, Lyft was scrappy in the beginning and just built whatever was needed to ship the product. It just organically grew from there. There was never a time to overhaul the systems.&#8221; <\/p>\n<p>\u2014Vicki Cheung, Lyft<\/p><\/blockquote>\n<p>Another issue that became prominent as the startup expanded was the need to separate the infrastructure and application layers. Product engineers still need to think about infrastructure code when they should be focused on their apps as explained by <a href=\"https:\/\/www.linkedin.com\/in\/lita-cho-511b033\/\" rel=\"noopener noreferrer\" target=\"_blank\">Lita Cho<\/a> and <a href=\"https:\/\/www.linkedin.com\/in\/josenino\/\" rel=\"noopener noreferrer\" target=\"_blank\">Jose Nino<\/a>, Senior Software Engineers at Lyft, during KubeCon North America 2018.<\/p>\n<div id=\"attachment_60891\" style=\"width: 110px\" class=\"wp-caption alignright\"><img decoding=\"async\" aria-describedby=\"caption-attachment-60891\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lita-Cho-150x150.jpg\" alt=\"\" width=\"100\" class=\"size-thumbnail wp-image-60891\" \/><p id=\"caption-attachment-60891\" class=\"wp-caption-text\"><small>Lita Cho<\/small><\/p><\/div>\n<blockquote><p>&#8220;We want product engineers to think just about their apps and minimize the infrastructure work they need to do, reducing the cognitive load.&#8221; <\/p>\n<p>\u2014Lita Cho, Lyft<\/p><\/blockquote>\n<p>To address both concerns, the company began migrating to Kubernetes in 2018.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Creating_a_hybrid_environment\"><\/span>Creating a hybrid environment<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Initially, Lyft was using virtual machines on Amazon Web Services (AWS). To manage its network, the company created <a href=\"https:\/\/www.envoyproxy.io\/\" rel=\"noopener noreferrer\" target=\"_blank\">Envoy<\/a>, a distributed proxy for services and applications.<\/p>\n<p>In order to migrate to Kubernetes without any downtime, the company set two goals:<\/p>\n<ul>\n<li style=\"margin-bottom: 6px;\">The new infrastructure had to be transparent for product engineers.<\/li>\n<li>The new infrastructure had to be reliable, allowing incremental rollouts and rollbacks in case of errors.<\/li>\n<\/ul>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Topology-Before-Migration-Kubernetes-Envoy-v2.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Topology-Before-Migration-Kubernetes-Envoy-v2-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60970\" \/><\/a><small>Architecture at Lyft prior to migration (<a href=\"https:\/\/www.youtube.com\/watch?v=xQxIoy50Mzo\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<div id=\"attachment_60890\" style=\"width: 110px\" class=\"wp-caption alignright\"><img decoding=\"async\" aria-describedby=\"caption-attachment-60890\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Jose-Nino-150x150.jpg\" alt=\"\" width=\"100\" class=\"size-thumbnail wp-image-60890\" \/><p id=\"caption-attachment-60890\" class=\"wp-caption-text\"><small>Jose Nino<\/small><\/p><\/div>\n<blockquote><p>&#8220;We want to build a system that makes it easy for product engineers to operate the network regardless of what may be behind the scenes. We want to make this new infrastructure accessible to use, while at the same time maintaining Lyft&#8217;s real-time system reliable.&#8221; <\/p>\n<p>\u2014Jose Nino, Lyft<\/p><\/blockquote>\n<p>During the migration, Envoy played a crucial role, serving as a sidecar to every service. This abstracted the network and enabled resources, wherever they may be located, to be routed to the service making the request. Specifically, Envoy xDS, which consist of discovery services and APIs, can dynamically configure the data plane from a centralized server.<\/p>\n<blockquote><p>&#8220;The xDS APIs can configure the data plane from a central xDS control plane, separating the concerns between configuration and the processing requests.&#8221; \u2014Jose Nino, Lyft<\/p><\/blockquote>\n<p>To better understand this abstraction, Envoy xDS control plane pulls data from Lyft&#8217;s legacy discovery service and uses Kubernetes watch APIs to retrieve IP addresses of pods. All this data is then converted and configured into the Envoy sidecars. This creates a hybrid environment, where Kubernetes services can interact with legacy services.<\/p>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Hybrid-Service-Discovery-Kubernetes-Envoy-v2.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Hybrid-Service-Discovery-Kubernetes-Envoy-v2-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60971\" \/><\/a><small>Envoy routes traffic for Kubernetes and the legacy stack (<a href=\"https:\/\/www.youtube.com\/watch?v=xQxIoy50Mzo\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<blockquote><p>&#8220;Because we are able to send out all those IP addresses with the Envoy sidecars, they now form a mesh. There&#8217;s now no difference between legacy and Kubernetes. Services can be split between the two, and we can still serve traffic to each other.&#8221; \u2014Lita Cho, Lyft<\/p><\/blockquote>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Migration_roadmap_and_results\"><\/span>Migration roadmap and results<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div id=\"attachment_60892\" style=\"width: 110px\" class=\"wp-caption alignright\"><img decoding=\"async\" aria-describedby=\"caption-attachment-60892\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Paul-Fischer-150x150.jpg\" alt=\"\" width=\"100\" class=\"size-thumbnail wp-image-60892\" \/><p id=\"caption-attachment-60892\" class=\"wp-caption-text\"><small>Paul Fischer<\/small><\/p><\/div>\n<p>While Lyft began migrating its workloads to Kubernetes in 2018, the company was already investigating the container orchestration system in 2017 noted <a href=\"https:\/\/www.linkedin.com\/in\/paulnivin\/\" rel=\"noopener noreferrer\" target=\"_blank\">Paul Fisher<\/a> and <a href=\"https:\/\/www.linkedin.com\/in\/ashleykasim\/\" rel=\"noopener noreferrer\" target=\"_blank\">Ashley Kasim<\/a>, Staff Software Engineers at Lyft, during KubeCon Europe 2020. They realized the need for a networking platform that could operate both the legacy stack and Kubernetes. This led to the creation of <a href=\"https:\/\/github.com\/lyft\/cni-ipvlan-vpc-k8s\" rel=\"noopener noreferrer\" target=\"_blank\">cni-ipvlan-vpc-k8s<\/a>, Lyft&#8217;s virtual private cloud (VPC) container networking interface (CNI) stack, which was later open sourced.<\/p>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Migration-Timeline-Kubernetes-Envoy-KubeCon-v2.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Migration-Timeline-Kubernetes-Envoy-KubeCon-v2-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60972\" \/><\/a><small>Lyft&#8217;s migration timeline (<a href=\"https:\/\/static.sched.com\/hosted_files\/kccnceu20\/9f\/kubecon-eu-2020-multicluster-lyft.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<p>In 2018, Lyft batch and machine learning workloads were migrated to Kubernetes. Next were the company&#8217;s stateless services in 2019 and stateful services in 2020.<\/p>\n<p>Currently, Lyft has multiple clusters for its different services. The company has a cluster with over 5,000 pods running machine learning workloads. <a href=\"https:\/\/github.com\/sbueringer\/kubecon-slides\/blob\/master\/slides\/2019-kubecon-na\/Running%20Large-Scale%20Stateful%20Workloads%20On%20Kubernetes%20at%20Lyft%20-%20Surinder%20Singh%20%26%20Anmol%20Khurana%2C%20Lyft%20-%20Stateful-workloads-lyft-kubecon.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">Flyte<\/a>, a distributed processing platform for machine learning and data workflows, which was also open sourced in 2019, runs on another cluster with more than 10,000 pods.<\/p>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Flyte-Architecture.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Flyte-Architecture-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60987\" \/><\/a><small>Flyte&#8217;s high-level architecture (<a href=\"https:\/\/github.com\/sbueringer\/kubecon-slides\/blob\/master\/slides\/2019-kubecon-na\/Running%20Large-Scale%20Stateful%20Workloads%20On%20Kubernetes%20at%20Lyft%20-%20Surinder%20Singh%20%26%20Anmol%20Khurana%2C%20Lyft%20-%20Stateful-workloads-lyft-kubecon.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<p>Lyft&#8217;s largest cluster is dedicated to its ridesharing service. This consists of over 600 stateless microservices and about 300,000 containers running on a single Envoy mesh. The cluster has around 30,000 pods that automatically scale up or down.<\/p>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Environment-KubeCon-v2.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Environment-KubeCon-v2-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60973\" \/><\/a><small>Lyft&#8217;s Kubernetes stack (<a href=\"https:\/\/static.sched.com\/hosted_files\/kccnceu20\/9f\/kubecon-eu-2020-multicluster-lyft.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<blockquote><p>&#8220;The reason we can have this multicluster environment that works is that we are keeping it as simple as possible. There are no overlay networks and NAT. There&#8217;s very limited use of ingress. We don&#8217;t use kube-proxy. Pods can communicate with pods in other clusters, and Envoy binds it all together.&#8221;<br \/>\n\u2014Paul Fisher, Lyft<\/p><\/blockquote>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Managing_multiple_clusters\"><\/span>Managing multiple clusters<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Lyft uses EnvoyManager, a custom control plane, to operate its multicluster environment. According to Ashley, EnvoyManager is mounted on top of <a href=\"https:\/\/github.com\/lyft\/go-control-plane\" rel=\"noopener noreferrer\" target=\"_blank\">go-control-plane<\/a>, and it uses Kubernetes informers and Lyft&#8217;s in-house discovery implementation to monitor pods and bridge the mesh across clusters.<\/p>\n<p>Lyft service pods run on an Envoy sidecar container. The sidecars then use DNS to locate EnvoyManager, which is run locally on the cluster as a <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/service\/#headless-services\" rel=\"noopener noreferrer\" target=\"_blank\">headless service<\/a>. The services run pods on end clusters that are redundant, making them resilient to outages.<\/p>\n<p><center><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-EnvoyManager-Deployment-KubeCon-v2.png\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-EnvoyManager-Deployment-KubeCon-v2-1024x576.png\" alt=\"\" width=\"640\" class=\"aligncenter size-large wp-image-60974\" \/><\/a><small>Multiple clusters communicating via EnvoyManager (<a href=\"https:\/\/static.sched.com\/hosted_files\/kccnceu20\/9f\/kubecon-eu-2020-multicluster-lyft.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">Image credit<\/a>)<\/small><\/center><\/p>\n<p>EnvoyManager is fault-tolerant as its multiple independent replicas are deployed across end clusters. This also ensures scaling depending on the load.<\/p>\n<div id=\"attachment_60893\" style=\"width: 110px\" class=\"wp-caption alignright\"><img decoding=\"async\" aria-describedby=\"caption-attachment-60893\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Ashley-Kasim-150x150.jpg\" alt=\"\" width=\"100\" class=\"size-thumbnail wp-image-60893\" \/><p id=\"caption-attachment-60893\" class=\"wp-caption-text\"><small>Ashley Kasim<\/small><\/p><\/div>\n<blockquote><p>&#8220;Each EnvoyManager talks to API servers of Kubernetes clusters. This means that each EnvoyManager has a complete picture of the whole state of the network, even though each Kubernetes cluster remains completely independent.&#8221; <\/p>\n<p>\u2014Ashley Kasim, Lyft<\/p><\/blockquote>\n<p>Lyft is currently working on improving EnvoyManager. With the beta introduction of <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/endpoint-slices\/\" rel=\"noopener noreferrer\" target=\"_blank\">EndpointSlices<\/a> in Kubernetes v1.17, the company is investigating a scalable implementation for tracking a large number of networking points. Lyft&#8217;s engineering team is also considering switching from a pod informer to an endpoint informer to resolve a few control plane issues.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Want_details_Watch_the_videos\"><\/span>Want details? Watch the videos!<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><small>In this video, <a href=\"https:\/\/www.linkedin.com\/in\/lita-cho-511b033\/\" rel=\"noopener noreferrer\" target=\"_blank\">Lita Cho<\/a> and <a href=\"https:\/\/www.linkedin.com\/in\/josenino\/\" rel=\"noopener noreferrer\" target=\"_blank\">Jose Nino<\/a> explain how Lyft utilizes Envoy to split traffic between the legacy and Kubernetes infrastructures.<\/small><\/p>\n<p><center><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/xQxIoy50Mzo\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/center><\/p>\n<p>&nbsp;<br \/>\n<small>In this next video, <a href=\"https:\/\/www.linkedin.com\/in\/paulnivin\/\" rel=\"noopener noreferrer\" target=\"_blank\">Paul Fisher<\/a> and <a href=\"https:\/\/www.linkedin.com\/in\/ashleykasim\/\" rel=\"noopener noreferrer\" target=\"_blank\">Ashley Kasim<\/a> discuss how Lyft manages its multicluster Kubernetes deployment.<\/small><br \/>\n&nbsp;<br \/>\n<center><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/7aMgS1uRU5s\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/center><!--more--><\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Further_reading\"><\/span>Further reading<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/the-ways-to-streamline-apps-and-ops-on-azure-kubernetes-service\/\">The Ways to Streamline Apps and Ops on Azure Kubernetes Service<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/maturity-of-app-deployments-on-kubernetes-from-manual-to-automated-ops\/\">Maturity of App Deployments on Kubernetes: From Manual to Automated Ops<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/running-stateful-apps-on-kubernetes-with-statefulsets\/\">Considerations for Running Stateful Apps on Kubernetes<\/a><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"About_the_experts\"><\/span>About the experts<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Vicki-Cheung-bio-150x150.png\" alt=\"\" width=\"100\" class=\"alignright size-thumbnail wp-image-60896\" \/><\/p>\n<p><small><a href=\"https:\/\/www.linkedin.com\/in\/vickicheung\/\" rel=\"noopener noreferrer\" target=\"_blank\">Vicki Cheung<\/a> is a cofounder and CTO at Stealth. Previously, she was Engineering Manager at Lyft, where she helped drive the company-wide Kubernetes migration. Vicki managed the team responsible for the compute fleet of Lyft&#8217;s workloads. Prior to Lyft, she was Head of Infrastructure and Founding Engineer at OpenAI, where Vicki and her team built out a Kubernetes-based deep learning infrastructure. Additionally, she spent time on managing, recruiting, designing engineering interviews, as well as leading diversity and inclusion efforts.<\/small><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Jose-Nino-bio-150x150.png\" alt=\"\" width=\"100\" class=\"alignright size-thumbnail wp-image-60897\" \/><\/p>\n<p><small><a href=\"https:\/\/www.linkedin.com\/in\/josenino\/\" rel=\"noopener noreferrer\" target=\"_blank\">Jose Nino<\/a> is Staff Software Engineer at Lyft. He has worked on the networking team for 2+ years building out the infrastructure that enabled the company to scale technically and socially, as it developed and rolled out an Envoy-based service-oriented architecture. Jose was instrumental in building control plane technologies and resilience improvements that allowed Lyft to grow to a 2,000+ engineer organization with 500+ microservices.<\/small><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lita-Cho-bio-150x150.png\" alt=\"\" width=\"100\" class=\"alignright size-thumbnail wp-image-60898\" \/><\/p>\n<p><small><a href=\"https:\/\/www.linkedin.com\/in\/lita-cho-511b033\/\" rel=\"noopener noreferrer\" target=\"_blank\">Lita Cho<\/a> is a cofounder and CTO at Moment Technologies. Previously, she was Staff Software Engineer at Lyft, where Lita worked on the networking team, building out the service mesh to handle both Kubernetes and legacy systems. She also maintained the tracing infrastructure at Lyft. Before that, Lita worked on building out the API infrastructure using Protocol Buffers, creating systems that would generate code and bring type safety to Lyft&#8217;s polyglot microservice architecture.<\/small><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Paul-Fischer-bio-150x150.png\" alt=\"\" width=\"100\" class=\"alignright size-thumbnail wp-image-60899\" \/><\/p>\n<p><small><a href=\"https:\/\/www.linkedin.com\/in\/paulnivin\/\" rel=\"noopener noreferrer\" target=\"_blank\">Paul Fisher<\/a> is Staff Software Engineer at Lyft, where he works on all things infrastructure-related from monitoring software to the service provisioning stack. He\u2019s currently leading the Lyft migration to Kubernetes. Paul tends toward work that lies at the intersection of systems programming and scale. He has previously spoken at Monitorama and has worked on systems at Twitter, UC Berkeley, Red Hat, and the Free Software Foundation.<\/small><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Ashley-Kasim-bio-150x150.png\" alt=\"\" width=\"100\" class=\"alignright size-thumbnail wp-image-60900\" \/><\/p>\n<p><small><a href=\"https:\/\/www.linkedin.com\/in\/ashleykasim\/\" rel=\"noopener noreferrer\" target=\"_blank\">Ashley Kasim<\/a> is Staff Software Engineer at Lyft. She is part of the compute infrastructure team at the company. Ashley is currently working on building out a resilient Kubernetes infrastructure that enables Lyft to run at scale.<\/small><\/p>\n<hr \/>\n<p><center><small>This blog post was written by <a href=\"https:\/\/www.altoros.com\/blog\/author\/carlo\/\">Carlo Gutierrez<\/a>, edited by <a href=\"https:\/\/www.altoros.com\/blog\/author\/sophie.turol\/\">Sophia Turol<\/a> and <a href=\"https:\/\/www.altoros.com\/blog\/author\/alex\/\">Alex Khizhniak<\/a>.<\/small><\/center><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scaling too fast<\/p>\n<p>Lyft is one of the largest ridesharing companies in the US, operating in all 50 states, as well as in Toronto, Ottawa, and Vancouver. Headquartered in San Francisco, the company reported a total revenue of $2.3 billion in 2020.<\/p>\n<p>A few years after its founding in 2012, Lyft had [&#8230;]<\/p>\n","protected":false},"author":32,"featured_media":60995,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":"","_links_to":"","_links_to_target":""},"categories":[26],"tags":[873,815,912],"class_list":["post-60779","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-use-cases","tag-cloud-native","tag-digital-transformation","tag-kubernetes"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.6 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment | Altoros<\/title>\n<meta name=\"description\" content=\"With a hybrid Kubernetes environment, the company automatically scales its ridesharing platform that consists of 600+ microservices on 30,000 pods.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment | Altoros\" \/>\n<meta property=\"og:description\" content=\"Scaling too fast Lyft is one of the largest ridesharing companies in the US, operating in all 50 states, as well as in Toronto, Ottawa, and Vancouver. Headquartered in San Francisco, the company reported a total revenue of $2.3 billion in 2020. A few years after its founding in 2012, Lyft had [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/\" \/>\n<meta property=\"og:site_name\" content=\"Altoros\" \/>\n<meta property=\"article:published_time\" content=\"2021-03-31T15:42:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-03-21T10:57:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"576\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/gif\" \/>\n<meta name=\"author\" content=\"Carlo Gutierrez\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Carlo Gutierrez\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/\",\"url\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/\",\"name\":\"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment | Altoros\",\"isPartOf\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif\",\"datePublished\":\"2021-03-31T15:42:25+00:00\",\"dateModified\":\"2022-03-21T10:57:28+00:00\",\"author\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/833e109f77de753b2b472dca0236b442\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#primaryimage\",\"url\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif\",\"contentUrl\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif\",\"width\":1024,\"height\":576},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.altoros.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.altoros.com\/blog\/#website\",\"url\":\"https:\/\/www.altoros.com\/blog\/\",\"name\":\"Altoros\",\"description\":\"Insight\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.altoros.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/833e109f77de753b2b472dca0236b442\",\"name\":\"Carlo Gutierrez\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/02\/CG_portrait-2-96x96.jpg\",\"contentUrl\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/02\/CG_portrait-2-96x96.jpg\",\"caption\":\"Carlo Gutierrez\"},\"description\":\"Carlo Gutierrez is a Technical Writer at Altoros. As part of the editorial team, his focus has been on emerging technologies such as Cloud Foundry, Kubernetes, blockchain, and the Internet of Things. Prior to Altoros, he primarily wrote about enterprise and consumer technology. Carlo has over 12 years of experience in the publishing industry. Previously, he served as an Editor for PC World Philippines and Questex Asia, as well as a Designer for Tropa Entertainment.\",\"url\":\"https:\/\/www.altoros.com\/blog\/author\/carlo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment | Altoros","description":"With a hybrid Kubernetes environment, the company automatically scales its ridesharing platform that consists of 600+ microservices on 30,000 pods.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/","og_locale":"en_US","og_type":"article","og_title":"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment | Altoros","og_description":"Scaling too fast Lyft is one of the largest ridesharing companies in the US, operating in all 50 states, as well as in Toronto, Ottawa, and Vancouver. Headquartered in San Francisco, the company reported a total revenue of $2.3 billion in 2020. A few years after its founding in 2012, Lyft had [...]","og_url":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/","og_site_name":"Altoros","article_published_time":"2021-03-31T15:42:25+00:00","article_modified_time":"2022-03-21T10:57:28+00:00","og_image":[{"width":1024,"height":576,"url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif","type":"image\/gif"}],"author":"Carlo Gutierrez","twitter_misc":{"Written by":"Carlo Gutierrez","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/","url":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/","name":"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment | Altoros","isPartOf":{"@id":"https:\/\/www.altoros.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#primaryimage"},"image":{"@id":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#primaryimage"},"thumbnailUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif","datePublished":"2021-03-31T15:42:25+00:00","dateModified":"2022-03-21T10:57:28+00:00","author":{"@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/833e109f77de753b2b472dca0236b442"},"breadcrumb":{"@id":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#primaryimage","url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif","contentUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/03\/Lyft-Kubernetes-Digital-Transformation-Envoy-Hybrid-Environment-KubeCon-Migration-cloud-native-v2.gif","width":1024,"height":576},{"@type":"BreadcrumbList","@id":"https:\/\/www.altoros.com\/blog\/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.altoros.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment"}]},{"@type":"WebSite","@id":"https:\/\/www.altoros.com\/blog\/#website","url":"https:\/\/www.altoros.com\/blog\/","name":"Altoros","description":"Insight","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.altoros.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/833e109f77de753b2b472dca0236b442","name":"Carlo Gutierrez","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/02\/CG_portrait-2-96x96.jpg","contentUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2021\/02\/CG_portrait-2-96x96.jpg","caption":"Carlo Gutierrez"},"description":"Carlo Gutierrez is a Technical Writer at Altoros. As part of the editorial team, his focus has been on emerging technologies such as Cloud Foundry, Kubernetes, blockchain, and the Internet of Things. Prior to Altoros, he primarily wrote about enterprise and consumer technology. Carlo has over 12 years of experience in the publishing industry. Previously, he served as an Editor for PC World Philippines and Questex Asia, as well as a Designer for Tropa Entertainment.","url":"https:\/\/www.altoros.com\/blog\/author\/carlo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/60779","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/comments?post=60779"}],"version-history":[{"count":106,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/60779\/revisions"}],"predecessor-version":[{"id":66549,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/60779\/revisions\/66549"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/media\/60995"}],"wp:attachment":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/media?parent=60779"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/categories?post=60779"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/tags?post=60779"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}