Benchmarks and Research
During the recent years, our R&D engineers performed multiple benchmarks of emerging technologies (such as NoSQL DBs, Hadoop, Cloud Foundry PaaS, etc.). Some of our studies were published by CIO.com, NetworkWorld, ComputerWorld, TechWorld, and other tech magazines. Download these white papers to get the results of our findings on performance, supported features, optimization, etc.
This technical report provides an in-depth analysis of three NoSQL databases: Couchbase Server v6.6.0, MongoDB v4.2.11, and DataStax Enterprise v6.8.3 (Cassandra). The study evaluates the throughput and latency of each database under four different workloads and three cluster configurations. For consistency, the comparison was conducted using the Yahoo! Cloud Serving Benchmark. The paper includes four comparative diagrams and seven descriptive tables to support the evaluation results.
This technical report provides an in-depth analysis of the three NoSQL systems: Couchbase Cloud, MongoDB Atlas, and Amazon DynamoDB. The comparison analyzes the throughput and latency of each database as a service under four different workloads and three cluster configurations. The evaluation was conducted using the Yahoo! Cloud Serving Benchmark as a default tool for consistency. The study includes seven tables and six diagrams to support the evaluation results.
This 11-page technical report compares four tools for testing a Kubernetes cluster—kube-bench, test-infra, Sonobuoy, and PowerfulSeal—across four criteria: a number of the covered scenarios, performance, usability, and customization. The study features four comparative tables, three code samples, and two figures to support the results.
This 11-page technical report compares three popular tools for Kubernetes cluster monitoring—Kube-prometheus, Elastic, and Zabbix—across five criteria: installation, alerting, ease of customization, ease of maintenance, and user interface. The study features five figures, five comparative tables, and four code samples to support the results.
This 12-page technical report overviews three popular Kubernetes deployment tools: Kubespray, kops, and conjure-up across six parameters: installation simplicity, configurability, scalability and distribution, supported cluster operations, performance, and debugging simplicity. To support the results, the document features six comparative tables and two code samples.
The healthcare industry aims to resolve multiple issues: expensive medical service, manual, paper-based workflows, lack of data privacy, etc. Beyond that, the COVID-19 pandemic triggered new challenges to deal with, including shortage of resources and lack of staff mobility. This research paper explores scenarios for improvement and exemplifies how cutting-edge technologies can help.
This 51-page technical report provides an analysis of the following database languages: MySQL, N1QL, and a MongoDB query. The analysis compares the three database languages against seven metrics across nine business scenarios. The paper features 17 figures, 11 tables, and 23 code samples, which illustrate the comparison in detail.
One of the most important traits of blockchain is transparency. However, some businesses that deal with sensitive information, such as those in finance, require a certain level of privacy and security. This paper explains how zero-knowledge proof / protocols can enable blockchains to continue verifying transactions while maintaining a degree of confidentiality.
Insurance is exposed to a variety of fraudulent schemes: from sharing an insurance plan after divorce to withholding critical information that may influence risk assessment. In this regard, blockchain can address some of the pain points found in the insurance industry. This study explains how the technology helps to improve security, mitigate fraud, automate record sharing between organizations, and cut operational costs. Finally, the paper covers some technical aspects of implementation and highlights scenarios where blockchain fits best.
This 12-page technical report provides guidelines for optimizing Apache Spark and two of its modules: Spark Core and Spark SQL. The study features the performance results of Apache Spark before and after optimization. The paper also includes four comparative diagrams and four descriptive tables, which illustrate performance differences in detail.
This technical report provides an in-depth analysis of the three leading NoSQL systems: Couchbase Server v6.0, MongoDB v4.0, and DataStax Enterprise v6.7 (Cassandra). The comparison evaluates the solutions across 20+ criteria, such as performance, availability, ease of installation and maintenance, data consistency, fault tolerance, replication, recovery, scalability, etc. The paper includes 26 scoring tables and 17 diagrams to support the evaluation results of the three systems.
This technical report compares three popular NoSQL systems: Couchbase Server v5.5, DataStax Enterprise v6 (Cassandra), and MongoDB v3.6. The evaluation includes performance results on 3 different cluster configurations and 4 varying workloads. The paper also provides comparative tables and diagrams which illustrate throughput and latencies measured in greater detail.
This technical report explores the architectural differences between the two popular NoSQL databases: Cassandra 2.1.x (DataStax Enterprise) and Couchbase Server 3.x. In detail, it describes how these discrepancies impact availability, scalability, and performance. The paper provides a detailed overview of topology, data model, partitioning, replication, caching, etc.
This report provides an in-depth analysis of the leading NoSQL systems: DataStax Enterprise v5.0 (Cassandra), MongoDB v3.4, and Couchbase Server v5.0. Approaching the databases from the perspective of architecture, development, and administration, the research compares the solutions across 20+ criteria: ease of installation/configuration, scalability, availability, maintenance, data consistency, fault tolerance, replication, recovery, etc. The benchmark features a scoring framework/template for evaluating and comparing NoSQL data stores for your particular scenario—depending on the weight of each criterion.
This 11-page technical paper describes how to manage secure Cloud Foundry deployments—distributed across multiple data centers. Providing a reference architecture, the guide explores how to create repeatable and secure Cloud Foundry deployments with related services using Vault, Concourse CI, and BOSH. Highlighting each of the components and pipelines involved into the workflow, the document also overviews dependencies between the elements of the architecture.
This 15-page benchmark demonstrates performance results of the most popular Ruby frameworks, template engines, Rack application servers, and Ruby ORM tools. All the Ruby frameworks and tools were tested in the production mode and with disabled logging to enable equal conditions.
This 15-page technical study compares five popular Ruby frameworks in terms of performance. The research paper includes performance results under four workloads: views (Slim), MySQL (Sequel), views and MySQL, and no views/DB. For each framework, similar test applications were built to find out how much time it takes to process 1,000 requests.
This 20-page technical research embodies performance evaluation of distributed training with TensorFlow under two scenarios: a multi-node and multi-GPU infrastructure configuration. The benchmark was carried out using the Inception architecture as a neural network model and the Camelyon16 data as a training set. To test training scalability of distributed TensorFlow running on an Amazon EC2 cluster, the g2.2xlarge and g2.8xlarge instance types were employed.
This 15-page research compares five popular deep learning tools—Caffe, Deeplearning4j, TensorFlow, Theano, and Torch—in terms of training performance and accuracy. When testing the frameworks, a fully connected neural network architecture was used for classifying digits from the MNIST data set. The paper also provides information on how speed and accuracy are affected by changes in the network “depth” and “width” (the data is available for the Tanh and ReLU activation functions).
This 25-page technical report evaluates performance of a sharded MongoDB cluster deployed over NetApp E-Series data storage. With 16 diagrams, the paper provides performance test results for SSD cache enabled / disabled and all-SSD scenarios, as well as analyses behavior on recovery after disk failure.
This 20-page guide provides step-by-step instructions on how to use Kibana—an open-source analytics / visualization framework—on the IBM Bluemix platform. In detail, the technical paper describes how to bind the Elasticsearch service to your app, configure Kibana, deploy it to Bluemix, and use for visualizing Elasticsearch data.
This paper questions if it is possible to boost performance of Hadoop calculations—using graphics processing units (GPU)—by up to 200 times. Exploring the bottlenecks of data flow between CPU, GPU, HDD, and memory, the study investigates the actual performance improvement that can be achieved. The article also reveals real-life performance results achieved by different projects and suggests a list of tools / libraries to use.
This 45-page technical paper compares two popular NoSQL offerings—MongoDB and Couchbase Server—exploring how architectural differences of these systems affect availability, scalability, and performance. The report also provides a detailed overview of topology, replication, caching, partitioning, memory, etc.
This 11-page research overviews NetApp E-Series storage systems in terms of performance, scalability, and availability—while running Couchbase Server. The paper compares NetApp E-Series to commodity servers with internal-attached storage (DAS), including performance and reliability characteristics under a number of workloads generated by a client-like benchmarking app. The paper also features a comparative table on DAS and E-Series performance, scalability, and availability.
This architect’s guide embraces the four major stages of Cloud Foundry adoption within an enterprise: CF evaluation, a proof of concept, POC assessment, and a PaaS rollout. Choosing an IaaS, configuring back-ups, integration, and having all the technology stack onboard is only a half of the way; you’ll also have to introduce cultural changes to get the best of the platform. Read this 60-page ultimate guide to learn what you need for a successful CF implementation—during all of the four adoption stages.
Nebula is a turn-key solution for distributed data center infrastructures that can help you to shorten deployment process to hours, not days. This 8-page tech study features a step-by-step guide on how to deploy the Cloud Foundry PaaS on Nebula. It includes detailed reference architecture for CF on Nebula, as well as infrastructure configuration, VM sizing recommendations, and the list of components necessary for a successful installation.
This 12-page research paper provides a scoring framework for evaluating API automation tools across 19+ technical criteria. It also includes a table that compares the functionality available in five popular API platforms: Apache UserGrid, WSO2 API Manager, Cumulocity, MuleSoft API Gateway, and StrongLoop Server. In addition, the study features step-by-step deployment guides for UserGrid and WSO2.
This 11-page benchmark compares 6 major Redis-as-a-Service products available on AWS: Redis Cloud Standard, Redis Cloud Cluster, ElastiCache, openredis, RedisGreen, and Redis To Go. While most performance tests focus on single get/set operations, this technical research demonstrates Redis behavior under 3 workload scenarios (simple, complex, and combined). The paper contains 3 comparative tables and 8 diagrams that show performance of each provider in close to real-life use cases.
This 15-page technical research explores the major pros and cons of microservices, compares them to monoliths, and demonstrates how a PaaS can help to overcome existing challenges. It includes 6 architecture diagrams, 3 comparative tables, and examples of running and scaling microservices applications on the Cloud Foundry PaaS. In addition, it provides a detailed comparison of how to operate microservices-based systems on an IaaS vs. PaaS and what benefits are available with each of these infrastructure options. Reference architecture and source code of a sample microservices application are also included.
This report provides an in-depth analysis of the leading NoSQL systems: Cassandra, MongoDB, and Couchbase. Unlike other comparisons that focus only on one or two dimensions, this research approaches the databases from 20+ angles, including performance, scalability, availability, ease of installation, maintenance, data consistency, fault tolerance, replication, recovery, etc. With 29 diagrams, this paper also features a scoring framework/template for evaluating and comparing NoSQL data stores for your particular use case—depending on the weight of each criterion.
Cloud computing remains one of the hottest topics in IT today given the promise of greatly improved efficiencies, significant cost savings, scalable infrastructure, high performance, and secured data storage. However, information on popular cloud systems is usually biased and full of advertising buzz, making it difficult to choose a cloud platform. This vendor-independent white paper contains a product-by-product comparison of the most popular cloud solutions (along with tips on bug-fixing) to help you select the best-fit product.
HP Moonshot is a low-power server designed for distributed computing and processing big data. This 12-page guide will help you to reap all the benefits of the Cloud Foundry PaaS—a platform for delivering cloud apps—deployed on top of Moonshot. You will learn how to install MaaS, assign roles to servers, deploy OpenStack, configure a new environment, set up routing, and provide high availability of apps on Cloud Foundry.
This 65-page benchmark compares the major Hadoop distributions: Cloudera, MapR, and Hortonworks. With 83 diagrams, it demonstrates how clusters of different size perform under 7 types of workloads. Download this study to learn which of the Hadoop distributions better suits your project needs and how to overcome limitations that may slow down your deployments.
Building a recommendation engine for a large online store involves a number of challenges, especially if you have to deal with huge data sets. In this white paper, our data scientist compares two approaches for implementing a Hadoop-based movie recommendation engine. Download the document to learn how generating association rules differs from clustering data, view the results of both implementations, and explore three ways to optimize the quality of movie recommendations.
Among all open source projects in the category known as Platform-as-a-Service, OpenShift and Cloud Foundry have amassed the strongest development communities. Having similar functionality and goals, both make it possible to write code in a variety of languages and deploy applications to public or private clouds. Both are evolving extremely fast. Still, few, if any, in-depth comparisons of OpenShift and Cloud Foundry exist. The purpose of this research is to provide a high-level overview of Cloud Foundry and OpenShift, side by side. With 2 tables and 2 diagrams, it describes their features, supported languages, frameworks, tools, architecture, history, etc.
The variety of NoSQL databases makes it difficult to select the best tool for a particular case. The unbiased investigations are rare—tests are often conducted by NoSQL vendors themselves. This benchmarking research is our vendor-independent analysis of four main NoSQL databases, based on performance measured under six different workloads.