Analyzing “Dark Data” for Traders Using IBM Watson and Bluemix

Information jeopardizing stock markets

Stock market trading has always been a high-risk, high-reward business, and the continued rise of social media has only reinforced it. There is an immense amount of money involved and there isn’t always enough time to analyze new information and react. Traders, who are able to react quickly, have the upper hand.

As an example, the tweet by US President Donald Trump last December had a huge financial impact on Lockheed Martin and Boeing.

Lockheed Martin stock after the tweet (Source)

This single 3-line post caused Lockheed Martin’s market cap drop by $1.2 billion while Boeing’s market cap rose by $500 million. Furthermore, Lockheed Martin’s shares dropped to $247.75 per share, while Boeing’s rose to $158.52 per share.

This brings us back to the problems in stock market trading today:

• There is too much information available 24/7 from both traditional and social media.
• There is not enough time for traders to gather, analyze, and react to new information.
• There is a significant amount of money at risk.

As a result, companies often gather more data than they are able to analyze.

Historical and “dark data” analysis

According to IBM, around 90% of data gathered from sensors and analog-to-digital conversions never gets used. It means the majority of such data simply becomes dark data—information that’s difficult to understand and analyze and is therefore not used in any decision making.

But can this be improved with technology?

In a stock market, a trader will typically analyze an asset by creating a time chart that compares the asset’s value over a period of time. To get more useful information, that asset’s data is compared to other assets to figure out its relative value. One of the ways to simplify this process is using tools like IBM Watson to create multi-dimensional charts that can compare relative assets.

Watson analytics chart (Source)

With this method, we can get a better understanding of stock patterns based on historical data. However, there’s a lot more that Watson can offer for trading.

Earlier last month, Watson entered the exchange trade fund (ETF) business. The Equbot with Watson AI Total US ETF was filed to the US Securities and Exchange Commission. According to the file, Equbot will use Watson to “conduct an objective, fundamental analysis of US-listed common stocks and real estate investment trusts based on up to ten years of historical data and apply that analysis to recent economic and news data.”

Spencer Fontein and Rob Williams of All Blue Solutions explain how dark data can be analyzed for stock market trading using IBM Watson and Bluemix. Based on the analysis of messages that appear on TV and social media, they believe that AI can be utilized to get a better understanding of trade position fluctuations as events transpire in real time.

Spencer Fontein and Rob Williams at InterConnect 2017

Insights from videos / TV

One of the most common methods of gathering news is from television. According to Spencer and Rob, IBM Watson can understand and parse data from TV and videos using the following steps:

• With a streaming television, convert MP4 streams with ffmpeg into 15 second videos and extract audio as WAV files.
• Using Watson Speech to Text (STT), send WAV files to Watson through cURL and save the response. (See STT in action.)
• With the AlchemyLanguage service, send, parse, and save the response.
• Send the response to the trading model, while IBM Watson decides if the response should affect the current position. Thus, one can adjust the model.

In this instance, Watson was used to gain insight from the news story of an oil rig accident in the Gulf of Mexico.

Oil rig accident in the Gulf of Mexico on April 1, 2015 (Source)

AlchemyLanguage analyzes the text file and thens extract such information as concepts, entities, sentiment, and keywords. This information can then be used to influence stock market trade decisions.

There’s also an IBM tool for analyzing videos, called OpenWhisk Dark Vision, which is very similar to the solution described by Spencer and Rob. Built on IBM Bluemix, it is a web app capable of analyzing video frames and audio, discovering what’s inside. After processing, the app automatically creates a summary for each video containing set of tags, personalities, and locations. This enables users to get an idea what each video contains without having to spend the time watching everything.

OpenWhisk Dark Vision web app (Source)

To analyze and extract information from videos, the app makes use both IBM Bluemix and Watson services:

• Visual Recognition for detecting scenes, objects, and faces
• Speech to Text for converting audio into text
• Natural Language Understanding for analyzing text and extracting metadata
• OpenWhisk for executing functions in response to event triggers
• Cloudant for storing everything in a database

Insights from social media and weather forecasts

With Twitter, Facebook, and Instagram, social media can now provide an even faster method of delivering news compared to traditional media. To analyze Twitter, for instance, one may use the following process and combination of services:

1. Capturing a tweet about market stock by Bluemix Insights for Twitter
2. Analyzing the tweet’s text by AlchemyLanguage
3. Analyzing the parsed data and formatting a response via Node-RED
4. Sending the response to the trading model
5. Adjusting the trading model based on the response received

Rob Williams

IBM Watson can additionally focus on relevant information by filtering through keywords, hashtags, topics, and specific sources.

Hazardous weather can have an impact on the stock market, as well. It can cause delays on shipping routes that will, in turn, incur financial penalties. By using IBM Weather Company Data for Bluemix, these delays can be predicted and accounted for. According to Rob and Spencer, a simplified process starts with a cURL call with a geolocation code (coordinates) to Weather Company Data.

Spencer Fontein

Weather Company Data will then return a response based on the coordinates given. This response is then sent to the trade model and will be used to adjust positions.

Current limitations

While Spencer and Rob are confident in IBM Watson and Bluemix’s ability to collect and analyze stock market data, there are also some limitations:

• Fake news can cause problems to the system.
• AlchemyLanguage is not always able to identify the correct keywords and entities.
• The trading models has to be revised as new variables are added.

Rob Williams

Insights for Twitter was retired and removed from the IBM Bluemix catalog on April 28, 2017. IBM suggests users to adopt Twitter’s Gnip API services instead.

AlchemyLanguage was deprecated on April 7, 2017 though support will persist until March 7, 2018. The service has already been absorbed by Watson’s Natural Language Understanding.

More tools for financial insights

Aside from the services already mentioned, we’ve come across other applications using Bluemix and Watson that focus on finance and trade analytics.

CloudTrader enables users to simulate an online stock trading system. Users are able to manage their portfolios, look up stocks, and buy or sell shares.

CloudTrader on Bluemix (Source)

Watson Investment Insights provides analysis on how news events can impact an investment portfolio. Through natural language processing and risk analytics, the application shows users how individual portfolios are affected by particular news articles.

Watson Investment Insights starter kit (Source)

In addition to that, here are a few more experimental services on Bluemix:

• Investment Portfolio enables users to store, update, and query investment portfolios and associated holdings using flexible object definitions.
• Instrument Analytics makes use of IBM Algorithmics software to price and compute analytics on securities.
• Predictive Market Scenarios enables users to understand “what if” scenarios by showing how market factors are affected by change.

Spencer Fontein

IBM created a number of starter kits for those who want to optimize their finace and trading activities with Watson services.

Real-time analytics and artificial intelligence are major trends these days. We’ve already seen what financial organizations are doing with the technology—from cognitive bank assistants to fraud detection based on behavioral analytics. Will we see a time when all human traders in the stock market are replaced by artificial intelligence? Probably not, but more assitance will definitely be provided to finance experts by analytical platforms like Watson.

Related videos

Further reading

• Digital Transformation in Banking: New Challenges for a New Era
• RBC Serves Millions of Customers Digitally—with IBM Bluemix, Blockchain, and AI
• IBM: The Time for IoT, AI, and Machine Learning Is Now

About the speakers

Rob Williams is Chief Technology Officer at All Blue Solutions Inc. He is responsible for worldwide technology operations, product development, and leadership and skills development. Rob works closely with customers, CTOs, and senior executives, ensuring delivery of best-of-breed solutions that meet changing needs. BlueMSP—a comprehensive managed service provider for IBM Software—was a project championed and continued to be led by Rob.

Spencer Fontein is Software Engineer at All Blue Solutions Inc. He develops and provides software solutions for mobile and web applications. Spencer uses a range of such IBM products as MobileFirst Platform, Bluemix, and Integration Bus in order to create the solutions his organization provides. He is a driven and passionate person, interested in learning new skills and applying them to IT development.