{"id":45527,"date":"2016-05-03T21:17:05","date_gmt":"2016-05-03T18:17:05","guid":{"rendered":"https:\/\/www.altoros.com\/blog\/?p=45527"},"modified":"2021-03-12T04:31:46","modified_gmt":"2021-03-12T01:31:46","slug":"using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix","status":"publish","type":"post","link":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/","title":{"rendered":"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#Description_of_the_Bluemix_Spark_starter\" >Description of the Bluemix Spark starter<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#Preparing_the_starter_for_Bluemix\" >Preparing the starter for Bluemix<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#Using_notebooks\" >Using notebooks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#Using_the_spark-submitsh_script\" >Using the spark-submit.sh script<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#Related_tutorials\" >Related tutorials:<\/a><\/li><\/ul><\/nav><\/div>\n<h3><span class=\"ez-toc-section\" id=\"Description_of_the_Bluemix_Spark_starter\"><\/span>Description of the Bluemix Spark starter<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/ibm-bluemix-apache-spark.png\" alt=\"ibm-bluemix-apache-spark\" width=\"100\" style=\"margin: 0px 0px 0px 15px;\" class=\"alignright size-full wp-image-45528\" \/><\/p>\n<p>When exploring Bluemix data and analytics services, I didn\u2019t find any starters or examples for how to use the IBM Apache Spark service. So, I decided to implement a simple starter on Scala and show how to work with the IBM service in Java\/Scala applications.<\/p>\n<p>My sample Bluemix Spark starter provides a simple solution for calculating the word count in a file. This file is stored in <a href=\"https:\/\/wiki.openstack.org\/wiki\/Swift\" target=\"_blank\" rel=\"noopener noreferrer\">Swift Object Storage<\/a>.<\/p>\n<p>At first, I wanted to deploy the starter in a Cloud Foundry container and have access to the Spark cluster from this starter, but the API usage of the Spark cluster in the IBM Analytics for Apache Spark service is not supported yet. For now, you can only use it through the Notebook UI interactively or the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">spark-submit.sh<\/code> script.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Preparing_the_starter_for_Bluemix\"><\/span>Preparing the starter for Bluemix<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>For running the Bluemix Spark starter, you need to create a new instance of the Apache Spark service from the Bluemix console. When setting up your Spark service instance, you can either create an Object Storage instance or connect to an existing Object Storage instance.<\/p>\n<p>If you haven&#8217;t already, sign up for <a href=\"https:\/\/cloud.ibm.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Bluemix<\/a>. In the Bluemix console, launch the IBM Analytics for Apache Spark service. For doing this, find Apache Spark in the <strong>Data and Analytics<\/strong> section of the Bluemix catalog, open the service, and then click <strong>Create<\/strong>.<\/p>\n<p><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/bluemix-apache-spark-catalog.png\" rel=\"attachment wp-att-10061\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/bluemix-apache-spark-catalog.png\" alt=\"bluemix-apache-spark-catalog\" width=\"640\" class=\"aligncenter size-full wp-image-45529\" \/><\/a><\/p>\n<p>Next, you need to clone <a href=\"https:\/\/github.com\/Altoros\/bluemix-spark-starter.git\" rel=\"noopener noreferrer\" target=\"_blank\">my repository<\/a> and change the following properties for Swift Object Storage:<\/p>\n<ul>\n<li><code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">auth.url<\/code> (is equal to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">auth_url<\/code> property from the Swift Object Storage service credentials)<\/li>\n<li><code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">username<\/code> (is equal to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">username<\/code> property from the Swift Object Storage service credentials)<\/li>\n<li><code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">password<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">apikey<\/code> (both are equal to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">password<\/code> property from the Swift Object Storage service credentials)<\/li>\n<li><code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">region<\/code> (is equal to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">region<\/code> property from the Swift Object Storage service credentials)<\/li>\n<li><code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">tenant<\/code> (is equal to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">tenant_id<\/code> property from the Apache Spark service credentials)<\/li>\n<\/ul>\n<p>To build an executable JAR file for <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">bluemix-spark-starter<\/code>, install Apache Maven and run the following command from the starter root:<\/p>\n<pre style=\"padding-left: 30px;\"><code>mvn clean, package<\/code><\/pre>\n<p><\/p>\n<p>After that, you can find the executable JAR file with the Spark job in the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">bluemix-spark-starter\/target<\/code> folder.<\/p>\n<p>To run analytics with IBM Analytics for Apache Spark, you can use Jupyter notebooks or the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">spark-submit.sh<\/code> script.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Using_notebooks\"><\/span>Using notebooks<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>From the <a href=\"https:\/\/console.bluemix.net\/docs\/services\/AnalyticsforApacheSpark\/index-gentopic2.html#genTopProcId3\" target=\"_blank\" rel=\"noopener noreferrer\">bluemix.net<\/a> website:<\/p>\n<blockquote>\n<p style=\"padding-left: 30px;\">\u201cJupyter Notebooks are a web-based environment for interactive computing. You can run small pieces of code that process your data and immediately view the results of your computation. Notebooks record how you worked with data, so you can understand exactly what was done with the data, even if time has passed since the notebooks were created. This enables you to reproduce computations reliably and share your findings with others.\u201d<\/p>\n<\/blockquote>\n<p>To run your analytics application using notebooks, create a new Scala notebook in your existing Bluemix Apache Spark service instance.<\/p>\n<p><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/bluemix-apache-spark-get-started.png\" rel=\"attachment wp-att-10062\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/bluemix-apache-spark-get-started.png\" alt=\"bluemix-apache-spark-get-started\" width=\"640\" class=\"aligncenter size-full wp-image-45530\" \/><\/a><\/p>\n<p>Run the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">AddJar<\/code> command to upload the executable JAR file to the IBM Analytics for Spark service in the first cell.<\/p>\n<p>For example:<\/p>\n<pre style=\"padding-left: 30px;\"><code>%AddJar https:\/\/github.com\/SergeiSidorov\/bluemix-spark-shell\/raw\/master\/com.altoros.bluemix.spark-1.0-SNAPSHOT.jar -f<\/code><\/pre>\n<p><\/p>\n<p>As mentioned in this <a href=\"https:\/\/developer.ibm.com\/clouddataservices\/start-developing-with-spark-and-notebooks\/\" target=\"_blank\" rel=\"noopener noreferrer\">IBM article<\/a>, the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">%<\/code> sign before <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">AddJar<\/code> is a special command that is currently available but may be deprecated in an upcoming release. The <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">-f<\/code> flag forces the download even if the file is already in the cache.<\/p>\n<p>After you deploy the JAR file, you can call APIs from within the notebook.<\/p>\n<p>For example, call functions from a notebook cell:<\/p>\n<pre style=\"padding-left: 30px;\"><code>val words = com.altoros.bluemix.spark.WordCount.countOfWord(sc)\r\nwords.foreach(p => println(\">>> word = \" + p._1 + \", count = \" + p._2))<\/code><\/pre>\n<p><\/p>\n<p>Final results in your Bluemix Jupyter Notebook look similar to this:<\/p>\n<p><a href=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/bluemix-jupyter-notebook.png\" rel=\"attachment wp-att-10063\"><img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/07\/bluemix-jupyter-notebook.png\" alt=\"bluemix-jupyter-notebook\" width=\"640\" class=\"aligncenter size-full wp-image-45531\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Using_the_spark-submitsh_script\"><\/span>Using the spark-submit.sh script<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Description of spark-submit.sh from <a href=\"https:\/\/console.bluemix.net\/docs\/services\/AnalyticsforApacheSpark\/index-gentopic3.html#using_spark-submit\" target=\"_blank\" rel=\"noopener noreferrer\">bluemix.net<\/a>:<\/p>\n<blockquote>\n<p style=\"padding-left: 30px;\">\u201c\u2014Runs the Apache Spark <em>spark-submit<\/em> command with the provided parameters.<br \/>\n\u2014Uploads JAR files and the application JAR file to the Spark cluster.<br \/>\n\u2014Submits a POST REST call to the Spark master with the path to the application file.<br \/>\n\u2014Periodically checks the Spark master for job status.<br \/>\n\u2014Downloads the job <em>stdout<\/em> and <em>stderr <\/em>files to your local file system.<\/p>\n<p style=\"padding-left: 30px;\">Restriction: Running the <em>spark-submit.sh<\/em> script is supported on Linux only.\u201d<\/p>\n<\/blockquote>\n<p>To use the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">spark-submit.sh<\/code> script for running your Spark application, do the following:<\/p>\n<ol>\n<li style=\"margin-bottom: 12px\">Create a folder where you plan to run the application and put the executable JAR file into this folder.<\/li>\n<li style=\"margin-bottom: 12px\">Launch an instance of the IBM Analytics for Apache Spark service and use the credentials from the <strong>Service Credentials<\/strong> page.<\/li>\n<li style=\"margin-bottom: 12px\">\n<p>Insert the Spark service credentials to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">vcap.json<\/code> file in the directory where you plan to run the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">spark-submit.sh<\/code> script.<\/p>\n<p>Here is an example from Bluemix documentation:<\/p>\n<pre style=\"padding-left: 30px;\"><code>{\r\n \"credentials\": {\r\n   \"tenant_id\": \"s1a9-7f40d9344b951f-42a9cf195d79\",\r\n   \"tenant_id_full\": \"b404510b-d17a-43d2-b1a9-7f40d9344b95_9c09a3cb-7178-43f9-9e1f-42a9cf195d79\",\r\n   \"cluster_master_url\": \"https:\/\/169.54.219.20:8443\/\",\r\n   \"instance_id\": \"b404510b-d17a-43d2-b1a9-7f40d9344b95\",\r\n   \"tenant_secret\": \"8b2d72ad-8ac5-4927-a90c-9ca9edfad124\",\r\n   \"plan\":\"ibm.SparkService.PayGoPersonal\"\r\n }\r\n}<\/code><\/pre>\n<\/li>\n<p><\/p>\n<li style=\"margin-bottom: 12px\">Download the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">spark-submit.sh<\/code> script from <a href=\"https:\/\/github.com\/Altoros\/bluemix-spark-starter.git\" target=\"_blank\" rel=\"noopener noreferrer\">this GitHub repository<\/a>.<\/li>\n<li style=\"margin-bottom: 12px\">\n<p>Run the script.<\/p>\n<p>For example:<\/p>\n<pre style=\"padding-left: 30px;\"><code>.\/spark-submit.sh \\\r\n--vcap .\/vcap.json \\\r\n--deploy-mode cluster \\\r\n--class com.altoros.bluemix.spark.WordCount \\\r\n--master https:\/\/169.54.219.20:8443 \\\r\ncom.altoros.bluemix.spark-1.0-SNAPSHOT.jar<\/code><\/pre>\n<\/li>\n<\/ol>\n<p><\/p>\n<p>Log files <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stdout<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stderr<\/code> are saved to your local file system. For more information about IBM Analytics for Apache Spark, see the <a href=\"https:\/\/cloud.ibm.com\/docs\/services\/AnalyticsforApacheSpark\/index.html\" target=\"_blank\" rel=\"noopener noreferrer\">service documentation<\/a>.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"Related_tutorials\"><\/span>Related tutorials:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/iot-platform-on-bluemix-experimenting-with-ibm-watson-and-gobot\/\">IoT Platform on Bluemix: Experimenting with IBM Watson and Gobot<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/deploying-kibana-to-ibm-bluemix-for-exploring-elasticsearch-data\/\">Deploying Kibana to IBM Bluemix for Exploring Elasticsearch Data<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/how-to-use-elastic-services-for-anomaly-detection-in-an-ibm-bluemix-app\/\">How to Use Elastic Services for Anomaly Detection in an IBM Bluemix App<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Description of the Bluemix Spark starter<\/p>\n<p>When exploring Bluemix data and analytics services, I didn\u2019t find any starters or examples for how to use the IBM Apache Spark service. So, I decided to implement a simple starter on Scala and show how to work with the IBM service in Java\/Scala applications.<\/p>\n<p>My [&#8230;]<\/p>\n","protected":false},"author":120,"featured_media":45536,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":"","_links_to":"","_links_to_target":""},"categories":[7],"tags":[873,187,748],"class_list":["post-45527","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news-and-opinion","tag-cloud-native","tag-ibm-bluemix","tag-machine-learning"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix | Altoros<\/title>\n<meta name=\"description\" content=\"This article demonstrates how to interact with IBM Analytics for Apache Spark through Jupiter Notebook and the spark-submit.sh script.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix | Altoros\" \/>\n<meta property=\"og:description\" content=\"Description of the Bluemix Spark starter When exploring Bluemix data and analytics services, I didn\u2019t find any starters or examples for how to use the IBM Apache Spark service. So, I decided to implement a simple starter on Scala and show how to work with the IBM service in Java\/Scala applications. My [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/\" \/>\n<meta property=\"og:site_name\" content=\"Altoros\" \/>\n<meta property=\"article:published_time\" content=\"2016-05-03T18:17:05+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-03-12T01:31:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/05\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif\" \/>\n\t<meta property=\"og:image:width\" content=\"640\" \/>\n\t<meta property=\"og:image:height\" content=\"299\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/gif\" \/>\n<meta name=\"author\" content=\"Siarhei Sidarau\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Siarhei Sidarau\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/\"},\"author\":{\"name\":\"Siarhei Sidarau\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/#\\\/schema\\\/person\\\/8b8187c5468c272bd951f988f3d75799\"},\"headline\":\"Using IBM Analytics for Apache Spark in Java\\\/Scala Apps on Bluemix\",\"datePublished\":\"2016-05-03T18:17:05+00:00\",\"dateModified\":\"2021-03-12T01:31:46+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/\"},\"wordCount\":790,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/05\\\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif\",\"keywords\":[\"Cloud-Native\",\"IBM Bluemix\",\"Machine Learning\"],\"articleSection\":[\"News\\\/Opinion\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/\",\"url\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/\",\"name\":\"Using IBM Analytics for Apache Spark in Java\\\/Scala Apps on Bluemix | Altoros\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/05\\\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif\",\"datePublished\":\"2016-05-03T18:17:05+00:00\",\"dateModified\":\"2021-03-12T01:31:46+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/#\\\/schema\\\/person\\\/8b8187c5468c272bd951f988f3d75799\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/05\\\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif\",\"contentUrl\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/05\\\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif\",\"width\":640,\"height\":299},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Using IBM Analytics for Apache Spark in Java\\\/Scala Apps on Bluemix\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/\",\"name\":\"Altoros\",\"description\":\"Insight\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/#\\\/schema\\\/person\\\/8b8187c5468c272bd951f988f3d75799\",\"name\":\"Siarhei Sidarau\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2018\\\/11\\\/1081.jpg\",\"url\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2018\\\/11\\\/1081.jpg\",\"contentUrl\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/wp-content\\\/uploads\\\/2018\\\/11\\\/1081.jpg\",\"caption\":\"Siarhei Sidarau\"},\"description\":\"Siarhei Sidarau is a software engineer at Altoros. He is an adept of Java, JavaScript, and SQL. Siarhei also has experience in database design and implementation, using MySQL, PostgreSQL, and Oracle.\",\"url\":\"https:\\\/\\\/www.altoros.com\\\/blog\\\/author\\\/siarhei-sidarau\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix | Altoros","description":"This article demonstrates how to interact with IBM Analytics for Apache Spark through Jupiter Notebook and the spark-submit.sh script.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/","og_locale":"en_US","og_type":"article","og_title":"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix | Altoros","og_description":"Description of the Bluemix Spark starter When exploring Bluemix data and analytics services, I didn\u2019t find any starters or examples for how to use the IBM Apache Spark service. So, I decided to implement a simple starter on Scala and show how to work with the IBM service in Java\/Scala applications. My [...]","og_url":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/","og_site_name":"Altoros","article_published_time":"2016-05-03T18:17:05+00:00","article_modified_time":"2021-03-12T01:31:46+00:00","og_image":[{"width":640,"height":299,"url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/05\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif","type":"image\/gif"}],"author":"Siarhei Sidarau","twitter_misc":{"Written by":"Siarhei Sidarau","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#article","isPartOf":{"@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/"},"author":{"name":"Siarhei Sidarau","@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/8b8187c5468c272bd951f988f3d75799"},"headline":"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix","datePublished":"2016-05-03T18:17:05+00:00","dateModified":"2021-03-12T01:31:46+00:00","mainEntityOfPage":{"@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/"},"wordCount":790,"commentCount":0,"image":{"@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#primaryimage"},"thumbnailUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/05\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif","keywords":["Cloud-Native","IBM Bluemix","Machine Learning"],"articleSection":["News\/Opinion"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/","url":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/","name":"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix | Altoros","isPartOf":{"@id":"https:\/\/www.altoros.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#primaryimage"},"image":{"@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#primaryimage"},"thumbnailUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/05\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif","datePublished":"2016-05-03T18:17:05+00:00","dateModified":"2021-03-12T01:31:46+00:00","author":{"@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/8b8187c5468c272bd951f988f3d75799"},"breadcrumb":{"@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#primaryimage","url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/05\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif","contentUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/05\/IBM-Analytics-for-Apache-Spark-in-Java-Scala-Apps-on-Bluemix.gif","width":640,"height":299},{"@type":"BreadcrumbList","@id":"https:\/\/www.altoros.com\/blog\/using-ibm-analytics-for-apache-spark-in-java-scala-apps-on-bluemix\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.altoros.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Using IBM Analytics for Apache Spark in Java\/Scala Apps on Bluemix"}]},{"@type":"WebSite","@id":"https:\/\/www.altoros.com\/blog\/#website","url":"https:\/\/www.altoros.com\/blog\/","name":"Altoros","description":"Insight","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.altoros.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/8b8187c5468c272bd951f988f3d75799","name":"Siarhei Sidarau","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2018\/11\/1081.jpg","url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2018\/11\/1081.jpg","contentUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2018\/11\/1081.jpg","caption":"Siarhei Sidarau"},"description":"Siarhei Sidarau is a software engineer at Altoros. He is an adept of Java, JavaScript, and SQL. Siarhei also has experience in database design and implementation, using MySQL, PostgreSQL, and Oracle.","url":"https:\/\/www.altoros.com\/blog\/author\/siarhei-sidarau\/"}]}},"_links":{"self":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/45527","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/users\/120"}],"replies":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/comments?post=45527"}],"version-history":[{"count":10,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/45527\/revisions"}],"predecessor-version":[{"id":60525,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/45527\/revisions\/60525"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/media\/45536"}],"wp:attachment":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/media?parent=45527"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/categories?post=45527"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/tags?post=45527"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}