Cloud Foundry for Pricing Optimization: GAP’s Experience
Achieve essential optimization of pricing, while saving time and computational resources to process large quantities of input data from multiple sources.
With a new tool set and the continuous integration and delivery (CI/CD) capability, the company is now able to change user interface overnight or push a feature into production in just five minutes.
- A DBaaS platform (e.g., Tesora) and a data virtualization platform (e.g., Denodo) can simplify data management.
- An operational intelligence solution, such as Splunk, will give you valuable insights into the inner workings of your applications.
- Use application performance monitoring tools to get a comprehensive view of what's going on across all of your systems.
- With the IPsec add-on, security of sensitive data within the Cloud Foundry is no longer an issue.
OpenStack, with plans to migrate to a public cloud—Azure or AWS
Gurobi, Spring Boot, RabbitMQ, MongoDB, Java, Jenkins CI, Splunk, Tesora, Denodo, Vault, Consul
Moving to Pivotal CF saves developers a lot of trouble with flowing new features into production. The company can now actually "deploy changes faster than people can consume them."
GAP Inc. is a global clothing and accessories retailer. The company has over 3,000 stores in operation across 52 countries and delivers its products to 214 countries. It is the 3rd largest international retailer with annual revenue of $16+ billion.
Donald and Doris Fisher, the company's founders, came up with the idea to open their first store as once they had difficult time finding jeans that would fit them.
Price optimization issues at GAP
At the Cloud Foundry Summit 2016, Philip Glebow mentioned localized promotional pricing as one of the GAP‘s initiatives in action. This marketing model relies on the principle of price adjustments, based on the local factors of demand and customer responses to changes in prices. The model has proven to be efficient enough to keep the inventory moving; however, this entails a bunch of issues for the company:
- Multiple parameters (including even weather conditions) need to be taken into account for the company to be able “to adjust the price and still move through the inventory.”
- Pretty sensitive deadlines—6,000 optimization instances to be completed in about 4 hours.
- Requires extensive computational resources (e.g., GAP uses about 90 Amazon’s c4.4xlarge instances).
Why cloud and Cloud Foundry?
With the above issues in mind, GAP started to look for a solution that would satisfy their requirements in a most efficient way. Eventually, the company opted for cloud services since those, as Philip Glebow explained, enable to:
- Vary costs—resources are scaled up or down to match the demand, and the company pays for just as much resources as it actually uses at a given moment
- Vary performance—make the process faster or slower as needed
- Eliminate the need for maintenance—the company does not have “to build a bunch of servers that sit around a data center.”
“The ability to build an elastic pricing solution that can scale on demand is very powerful.”
—Philip Glebow, GAP
Explaining the company’s choice of Pivotal Cloud Foundry, Philip Glebow pointed out the following advantages:
- Fast and easy integration and delivery of cloud services on a continuous basis with maximum flexibility
- Reduced processing time
- Very quick to scale
- Simplifies the infrastructure
- Imposes a set of useful patterns (e.g., microservices, 12-factor apps, etc.)
“If we need to change the algorithm—fine, we can run it through a unit test, it flows automatically to production.” —Philip Glebow, GAP
“Something that would normally, perhaps, take days or weeks—we can compress that into a cycle of few minutes.” —Philip Glebow, GAP
“Constrains somewhat what people can do, but in a way that I think is very empowering.”
—Philip Glebow, GAP
The adopted concept of microservices has brought a major change to the company’s development workflow, speeding up and automating deployment via a federated continuous integration pipeline.
Detailing the technical part of the Cloud Foundry deployment project, Philip Glebow described the system they actually built at the company. The system comprises:
- Pivotal Cloud Foundry as a PaaS
- a Java-based application that talks to MongoDB and RabbitMQ
- the Gurobi linear solver
- Spring Boot to work with applications
- a Spring actuator
Client access to the system is via a web-browser or the REST services.
The GAP and Cloud Foundry story actually started with Pivotal CF that was installed internally. Then, the company felt it has to have access to some other essential process components:
- the existing MongoDB database and RabbitMQ (both running on OpenStack)
- the Gurobi linear solver (running on Amazon)
That’s how they came to use Cloud Foundry and Spring config to manage their processes, as well as acquire and provision data.
Among major tasks of the existing system, the speaker mentioned managing large data quantities from multiples sources. To solve the task, the company employs the data virtualization technology that functions as an interface layer, enabling to access a variety of databases in use: Oracle, MySQL, MongoDB, etc.
“It allows us to expose an interface layer that can look very much the same across brands and, for us, that has been very powerful.” —Philip Glebow, GAP
At GAP, the workflow used to solve the optimization tasks is as follows:
- Creating cloud instances of the compute servers, scaling those up or down to the needs.
- The RabbitMQ component builds a central queue of optimization jobs, and then a dedicated group of workers deals with the queue, thus enabling to solve independent optimization problems in parallel.
- Processing data and reports.
“The process takes a good two minutes.” —Philip Glebow, GAP
What’s the price?
Just like any other project, building a Cloud Foundry-based system at GAP involved certain challenges:
- Experts in data science and optimization would use Python, MATLAB, or SAS—rather than Java chosen for the project.
- Essentially waterfall environment is unwilling to adopt the Agile development approach.
- Though Spring Cloud Config helped the company to accelerate scaling, simplify adding new instances, and cope with some other complex problems, its use revealed a number of inherent deficiencies:
- Hand encryption is inconvenient on an enterprise level.
- Git becomes a single soft spot, as its failure can cause an overall stop.
- One needs to mind issues with web hooks during Blue-Green deployment.
- The audit trail in Git missing information about the effective date of a committed change.
- There are more than 60 TB of logging data from numerous applications, which needs to be managed.
- Due to specifics of the operational process at GAP and the wide variety of databases used at the company, setting up data services for applications takes up much time and efforts.
- The company has strict security requirements concerning protection of personal data (e.g., names, addresses, credit card details, etc.).
Any project of as big a scale is sure a source of invaluable experience, both for the implementing company and anyone who follows its footsteps:
- Adopting a technology or a development approach that is different from what is widely accepted in the industry requires a major cultural change. The main thing here is try to be flexible, investigating into options and opportunities, and be prepared to take reasonable risks.
- Collaborating with a broader community can save time and resources in finding proper solutions to eliminate major deficiencies.
- Splunk can do the job of managing logging data just all right, while providing additional capabilities, such as alerting.
- Setting up data services could be made much simpler with a DBaaS platform, e.g., Tesora. Data virtualization technologies (e.g., Denodo data virtualization platform) can come in handy, too, especially if one has to deal with a lot of legacy systems and data centers.
- Additional use of specialized tools for application monitoring can help to quickly identify and solve any performance problems across the entire infrastructure—both in Cloud Foundry and legacy systems.
- Installing the IPsec add-on is enough to protect any sensitive information within Pivotal Cloud Foundry.
“We’re happy to make mistakes as long as the blast radius isn’t too large if it fails.”
—Philip Glebow, GAP
The company’s vision of its future is dominated by the idea of migrating to a public cloud—most probably, Microsoft Azure. Philip Glebow shared these plans at the SpringOne Platform conference in December 2017, with the following reasons mentioned as underlying the move:
- Better speed and a larger scale
- Geographic proximity of services
- New technologies to adopt, including machine learning
However, despite the advantages, there’s the financial side of the question that can’t but be taken into consideration.
“Some of these services are very cheap at the low end but, if you start to really scale them up and if you look at the cost modeling, I think it is really expensive, really fast,” —Philip Glebow, GAP
Therefore, the company will still have to “figure out the right way” when migrating to a public cloud. It’s interesting that another company, WMG, did a similar move with their Cloud Foundry deployment, migrating from OpenStack to AWS last year.
Want details? Watch the video!
Table of contents
- A Car App in 6 Months: Mercedes-Benz / Daimler Gains Pace with Cloud Foundry
- Warner Music Migrates Its Cloud Foundry Deployment from OpenStack to AWS
- The Dutch Gov Delivers an Alert System on Cloud Foundry in 7 Months
About the speaker
Philip Glebow is now the product architect for pricing in the planning domain at GAP Inc. He can boast of over 15 years of experience in software system design and cooperation with companies such as KPMG Consulting Inc., Blackrock Inc., and Barclays. Mr. Glebow holds a Bachelor of Science in Computer Engineering (Cal Poly San Luis Obispo) and a Master of Science in Bioinformatics (John Hopkins University)