Small number of companies actually have big data

Last week at the TDWI Big Data conference in San Diego I learned that there are relatively few companies that actually have big data.  Yes there are eBay (keynote address), Google, Facebook and other similar companies that have a large web presence who are actively utilizing big data solutions such as Hadoop.  However, many speakers and even some vendors agreed that only a few companies actually have data volumes, variety or velocity (the 3 V’s) that require a big data solution.  One speaker indicated that in his opinion, 85% of data warehouses are under 5Tb, which is considered “small” by TDWI standards.  It appears that Big Data is still in the “peak of inflated expectations” area of the Gartner hype cycle.  Companies need to compare their true data volumes, their available data sources (including traditional big data sources such as web logs, sensor data, etc…) and their business goals before they decide to go down the path of Big Data solutions.  The challenge in today’s world is identifying real world big data case studies (as there haven’t been that many implementations yet) where you can visualize and understand how big data architecture can enhance your understanding of corporate data and customer behavior.  Until this market becomes more mature and we escape extreme highs and lows of the hype cycle, we need to take the vendor comments with a grain of salt and objectively analyze where big data will truly add value to the corporation.  Avoid the hype, publish good data.