I would disagree with the premise that "big data" isn't being converted into valuable information. Modern advertising is being driven by exactly this ability. The problem lies not with the information, or lack thereof, but with the decision makers. Lack of vision at both the strategic and tactical levels of decision, fear of change, among other things are at the root of the problem.
I also agree. It really depends on what you do with the data. Even a small bit of data can mean a lot.
The other problem is that the people gathering the data and analyzing it are sometimes people with PhDs and not MBAs. We are talking about a very technical thing here and the industry is growing. Just give it some time and I guarantee it'll evolve. So how you ask? Time, that's how. Trial and error. The risk takers will lead the way.
To be honest, I do believe many people are wasting too much money on huge data warehouses. I firmly believe we can get the data we need to act on without going so crazy. We're all too impressed with size. I mean, maybe if you're trying to clone Dolly or are the NSA or something, you can't avoid all that data...But size doesn't matter. Results do.
For example, a 5 terabyte data set of Twitter messages from random people no one cares about talking about what they ate for breakfast? Useless.
I know, it's unfair...We've spent millions of dollars learning about breakfast foods from Twitter and Tony the Tiger's family is still starving. Yup, it happens.
Just give it time. Big data and data mining on the web is still in its infancy in terms of application.
I am not at all convinced that our ability to convert data to actionable decision making metrics is limited.
In fact, I do just that every day.
There are definitely limiting factors, but the ability to make use of the data is much stronger than people commonly believe.
For structured data you just need a good understanding of the data structures and their relationships. Unstructured data (tweets, statuses, etc.) is much more difficult but there are several technologies available and under development that tackle unstructured data specifically. This even includes some high end tools that monitor voice data.
If you have a specific data scenario in mind I can be more specific. If you'd like to arrange a call I'd be happy to provide more detail.
