What is Fog Computing?

Are you in the mood for a new technology buzzword?  CISCO's marketing department is rolling with the term 'Fog Computing'.

The Fog is an extension of the Cloud.  So in Fog Computing, data is stored closer to the devices we use to access the internet. The benefit is a reduction of bandwidth and latency.

CISCO routers already live in this grey area between out devices and big data providers. CISCO is adding Linux operating systems to some of their routers, and manage frequently accessed data with a distributed network of these router on the edge of the cloud.

by David Flectcher

Naturally several definitions of Fog Computing mention another buzzword, The Internet of Things. But the concept of caching data nearby isn't new.   My paper Smart Technology for Big Data talks about how Facebook caches images close to users based on how likely the images will be accessed.  Today's article Forget 'the Cloud'; 'the Fog' Is Tech's Future in the Wall Street Journal reluctantly embraces the new term.

Welcome to the Fog.

The Journal of Trading: Smart Technology for Big Data

Smart Technology for Big Data was published in the Winter edition of Journal of Trading.  You need to register to read them.  Here's the Abstract:

This article provides an underlying structure for managing the big data phenomenon. Innovations and tools fundamental to handling big data are highlighted, and we look at how these technologies are being implemented in the financial industry. 

See more at: http://www.iijournals.com/doi/abs/10.3905/jot.2013.9.1.057

Institutional Investor Journals: Big Data Article

UPDATE:   Smart Technology for Big Data was published in the Winter edition of Journal of Trading, so the links below no longer work. You can access the article here: Smart Technology for Big Data (You'll still need to register if you haven't)




My article Smart Technology for Big Data is published under advanced content at the Institutional Investor site.  You'll need to complete the free registration to read it. Enjoy!

The Year of the Yottabyte?

Big Data has been the big technology buzzword for a couple of years now.  So recently, as a nod to big data, the term yottabyte has  become a top technology buzzword. In my upcoming paper  "Smart Technology for Big Data" (for Institutional Investor Journals)  this chart introduces big data.

Exhibit 1

Bytes	Magnitude	Common usage	Example
gigabyte (billion)	109	computer RAM	New laptops have about 8GB RAM
terabyte (trillion)	1012	discussing computer hard drives sizes	NYSE produces about 1 TB  of information day
petabyte (quadrillion)	1015	total company storage space	Facebook’s largest Hadoop cluster contains 100 PB disk space
exabyte (quitillion)	1018	all the... in the world	Global internet traffic is 21 EB per month.
zetabyte (sextillion)	1021	future storage discussions	Total size of the internet is about 1 ZB.
yottabyte (septillion)	1024	speculation	Nearly infinite.

It's a favorite exhibit of those who have read the paper. Many had never heard of terms larger than a petabyte.  In the article, I mention that the term yottabyte is used in "speculation".  But is that speculation about to enter the realm of reality?  The recent article about Twitter adding security to impede surveillance mentions that the National Security Agency's datacenter in Bluffdale, Utah is "possibly cabable" of storing a yottabyte. We're still in speculation mode but for how long?  Will 2014 be the year of the Yottabyte?

Getting the Most Out of Your Big Data - Discovery by 1010data

[11-01-13] Getting the Most Out of Your Big Data - The Top 4 Trends - Discovery

Using Big Data to Discover New Customers

Still mass marketing potential customers with emails?  You may instead be driving them away.  Aside from being technologically dated,  unsolicited emails, far more than likely, never get read, leaving only a brief unprofessional impression.

MailChimp

In the article Making Big Data Accessible in MailChimp the founder of a company that makes money sending emails even warns about emails blasts.

Some of you might be thinking, “Great, now you can blast those people press releases!” If so, please unplug your computer now. I can think of no better way to lose a customer than to “blast” them with–well, anything. - Ben Chestnut MailChimp CEO

Instead of driving customers away,  MailChimp allows users to employ basic marketing techniques such as segmentation with their subscriber lists.  The Discovery feature uses big data techniques to parse millions of subscribers in minutes, and help find new customers.

Fun with Big Data Visualizations

Visualizations

Obtaining insights from Big Data requires new ways of looking at data. One of the most popular visualizations is the Tag Cloud.

The Tag Cloud is a creative way to convey lots of information about single articles, large websites, or even entire libraries, all on a single screen. Words are sized relative to the number of times they appear, and clustered together randomly to give a reader a different perspective. This Tag cloud contains the most common words on the AppTrain blog homepage.

A Mind Map

 is a Tag Cloud with relationships. Rather than random words, a Mind Map displays key concepts and how they relate to each other.

Document Driven Documents (d3.js) 

is a javascript library for building visualizations from any dataset. The libraries examples are an extensive collection of modern visualizations. Many alternate ways of adding dimensions to graphs are displayed on the example page including Voronoi Diagrams, Box Plots, Heat Maps, Bubble Charts,
Van Wijk Visualizations, Hamiltonian Graphs; Many of the examples are also interactive, changing the way the graph is viewed based on user input. Exploring the examples is truly mind expanding.

Predicting the Stock Market using Big Data

In the paper Quantifying Trading Behavior in Financial Markets Using Google Trends researchers Tobias Preis, Helen Susannah Moat and H. Eugene Stanley have shown that an increase in activity of certain search terms from Google Trends correlates with a decline in stock prices in the Dow Jones Industrial Average (DJIA). The authors then compared investment strategies to show that the search activity isn't just a correlation, but can be used as a valid predictor of market activity.

This graph shows the DJIA on the left, and color codes 3 week periods in the graph according to search frequencies of the word debt. Note that red weeks, like late October of 2008 correspond with declines in the DJIA.  So when lot's of people were searching for the work debt, the stock market went down.

The word Debt was the best performing term in the study.  Notably , it performed better than terms like nyse ,nasdq, and dow jones.

The researchers then compared a Google Trends investment strategy to a basic Buy and Hold strategy and a random Dow Jones strategy.

The results were remarkable.  The Google Trends strategy far exceeded the other strategies.

Kissell Research Group Conference

It was a tremendous honor to be asked to speak at the inaugural conference of  Kissell Research Group in New  York.  Dr. Robert Kissell is highly respected in financial circles, so the room at The Harvard Club was packed.  I spoke about Smart Technology for Big Data .  The conference was a huge success and we were mentioned in the latest newsletter from Institutional Investor Journals called The Voices of Influence. The Publisher Allison Adams wrote

"To put it in perspective, I was recently invited to attend an event on algorithmic trading hosted by Kissell Research Group. The last presentation by Michael Blake was on Smart Technology for Big Data." ...more

 Dr Kissell's presentation on the I-Star Impact Model and Dr Roberto Malamut's talk on Quantitative Trading were the highlights of the day, so trust me , I was riding on their coattails.

Smart Technology for Big Data

"Big Data" is so 1998

This 1998 SiliconGraphics ad from Black Enterprise magazine offers solutions for a "Big Data" world.  256GB of system memory on a server and 400 Terabytes of storage. Not bad for the 20th century. Or for this century.

The "Big Data" buzzword almost caught on in 1998, but it's sister buzzword, Data Mining won out.  In the  first chapter of "Predictive Data Mining: A Practical Guide is titled "Big Data" (also from 1998) the author Sholom M. Weiss asks "Is data mining a revolutionary new concept? or can we benefit from the may years of research on data analysis?”



Weiss goes on to say "While big data have the potential for better results, there is no guarantee that they are more predictive than small data" With all the hype around Big Data, it helps to get back to the origins of the term and realize that it's one of may interesting problems that experts in a variety of disciplines have been wrestling with for a long time.