Tuesday, January 7, 2014

Big Data Analytics by ICGX

The data revolution has only just begun. Everyone is talking about Big Data. Big Data grows up - Forbes Business opportunities is Big Data - INC. Big Data powers evolution decision making - WSJ How Big Data got so big - NYT Big Data is hot? Now what? - Forbes Businesses "freak out" over Big Data - Information Week 2012: The year of Big Data - WSJ The age of Big Data - NYT But it's not just hype. The world's data is doubling every 1.2 years. There are 7 billion people in the world. 5.1 billion of them owns cell phone. Each day, we send over 11 billion texts, watch over 2.8 billion YouTube videos and preform almost 5 billion google searches. And we're not just consuming it. We're creating it. We are data agents. We generate over 2.4 quintillion bytes everyday from consumer transactions, communication devices, online behavior, streaming service. In 2012, the world’s information totaled over 2 zetabyes. That’s 2 trillion gigabytes. By 2020, that number will be 35 trillion. We will need 10x more servers, 50x more data management, 75x more files to handle it all. If you're like most companies, you aren't ready. 80% of this new data is unstructured. It is too large, too complex, and too disorganized to be analyzed by traditional tools. There are 500K computer scientists yet only 30K mathematicians. We will fall short of the talent need to understand Big Data by at least 100K. To find opportunities in Big Data, we need new tools and new talent to mine this information and find value. We need Big Data Analytics. Big Data Analytics is more than technology. It’s a new way of thinking. It will help companies better understand customers, find hidden opportunities, even help our government better serve citizens and mitigate fraud. It will inspire hundreds, thousands and even millions of new startups. It will alter the landscape across virtually every industry and finally answer the questions looming over every CEO's head, "How can my business use Big Data?", "What problems can it solve?", "Who should be leading the charge, CIO, CMO, or Chief Data Scientists ?". In every revolution, there are opportunities that will be seized only by those armed with the right tools and right strategy. We are at the beginning of the Big Data Revolution.

Thursday, January 2, 2014

Running OpenTSDB on Amazon EC2

Although there are cheaper alternatives for production systems, It's easy enough to get The Open Time Series Database OpenTSDB running on an EC2 instance of Amazon Web Services.

  1. First you'll need to run HBase on EC2
  2. Make a data directory mkdir hbase_data
  3. vi hbase-0.94.13/conf/ hbase-site.xml
  4. Using vi update the hbase.rootdir property value to: file:///home/ec2-user/hbase-0.94.13/hbase-\${}/hbase
  5. sudo yum install git
  6. git clone git://
  7. sudo yum install automake
  8. yum install gnuplot
  9. cd opentsdb
  10. ./
  11. env COMPRESSION=NONE HBASE_HOME=path/to/hbase-0.94.X ./src/
  12. tsdtmp=${TMPDIR-'/tmp'}/tsd
  13. mkdir -p "$tsdtmp" 
  14. ./build/tsdb tsd --port=4242 --staticroot=build/staticroot --cachedir="$tsdtmp"
  15. In AWS, click on your EC2 instance, then click "Security Groups" at the bottom left.  Click on the default group, then click the "inbound" tab.  You can now open the ec2 port 4242. 
Your ip address on port 4242 will display the web UI for your instance of OpenTSDB:

  • Thursday, December 26, 2013

    Running HBase on Amazon EC2

    1. Create an Amazon Linux EC2 instance. 
    2. Log into your EC2 Instance using ssh.
    3. sudo yum install java-1.6.0-openjdk
    4. wget
    5. tar xfz hbase-*
    6. vi .bashrc
    7. Add this line at the bottom of the file JAVA_HOME=/usr/java/default
    8. sudo vi /etc/hosts
    9. Comment out the localhost line: #   localhost localhost.localdomain
    10. cd  hbase-*
    11. Start HBase ./bin/
    12. Check log files cat logs/hbase-*

    Tuesday, December 24, 2013

    The Journal of Trading: Smart Technology for Big Data

    Smart Technology for Big Data was published in the Winter edition of Journal of Trading.  You need to register to read them.  Here's the Abstract:

    This article provides an underlying structure for managing the big data phenomenon. Innovations and tools fundamental to handling big data are highlighted, and we look at how these technologies are being implemented in the financial industry. 

    See more at:

    Wednesday, December 18, 2013

    Institutional Investor Journals: Big Data Article

    UPDATE:   Smart Technology for Big Data was published in the Winter edition of Journal of Trading, so the links below no longer work. You can access the article here: Smart Technology for Big Data (You'll still need to register if you haven't)

    My article Smart Technology for Big Data is published under advanced content at the Institutional Investor site.  You'll need to complete the free registration to read it. Enjoy!

    Monday, December 9, 2013

    Grace Hopper

    Today's Google doodle is for Grace Hopper, a computer programmer who would have been 107 today.

    Grace was an admiral in the US Navy and worked on one of the first computers in 1945. She worked on the committee that created the COBOL programming language, and has a US Navy destroyer, the USS Hopper named after her.

    Grace's intelligence and charm shines through in this interview on David Letterman in 1986.  In a field where women are still underrepresented, it's tremendous that there are women like Grace Harper in computer science whom we cal all revere.

    Thursday, December 5, 2013

    Stay happy my friends

    Tech in Asia created this great infographic based on the results of the TD Small Business Happiness Index Survey.

    Interesting results, but there's no comparison to non-small business owners.  How happy are they?

    Tuesday, December 3, 2013

    The Year of the Yottabyte?

    Big Data has been the big technology buzzword for a couple of years now.  So recently, as a nod to big data, the term yottabyte has  become a top technology buzzword. In my upcoming paper  "Smart Technology for Big Data" (for Institutional Investor Journals)  this chart introduces big data.

    Exhibit 1

    Common usage
    gigabyte (billion)
    computer RAM
    New laptops have about 8GB RAM
    terabyte (trillion)
    discussing computer hard drives sizes
    NYSE produces about
    1 TB  of information day
    petabyte (quadrillion)

    total company storage space
    Facebook’s largest Hadoop cluster contains 100 PB disk space
    exabyte (quitillion)
    all the... in the world
    Global internet traffic is 21 EB per month.
    future storage discussions
    Total size of the internet is about 1 ZB.
    Nearly infinite.

    It's a favorite exhibit of those who have read the paper. Many had never heard of terms larger than a petabyte.  In the article, I mention that the term yottabyte is used in "speculation".  But is that speculation about to enter the realm of reality?  The recent article about Twitter adding security to impede surveillance mentions that the National Security Agency's datacenter in Bluffdale, Utah is "possibly cabable" of storing a yottabyte. We're still in speculation mode but for how long?  Will 2014 be the year of the Yottabyte?

    Friday, November 29, 2013

    Monday, November 11, 2013

    Java Swing Has to be Deprecated | Yakov Fain's Blog

    Yakov Fain explains how outdated the Java Swing API is .  I'd go further and add that a vast majority of  legacy Swing apps I've seen should we rewritten as Web Applications.  Even if an application is for a limited number of users behind a firewall, it will be much easier to  develop, maintain and use if it runs as a web application. Let the familiar web browser act as the user interface, and keep application clients code thin.

    Java Swing Has to be Deprecated — Every time I start teaching my new Java class I’m looking at the Swing units in the manual asking myself, “Why my students need to know Swing framework?” Well, I need to teach them how to program GUI, event listeners, asynchronous worker threads and event loop that are pretty much the same in every programming […]

    Friday, October 25, 2013

    Business Training Apps by Train OnTheMove

    Top quality business mobile learning applications. From Management and Sales, to Leadership and Project Management. Yes, we have a business training app for that. Available at the App Store Get it at the BlackBerry App World