How big data can evolve if our overall ability to convert data into really valuable info that then can drive effective decision making is so limited?

I also agree. It really depends on what you do with the data. Even a small bit of data can mean a lot.

The other problem is that the people gathering the data and analyzing it are sometimes people with PhDs and not MBAs. We are talking about a very technical thing here and the industry is growing. Just give it some time and I guarantee it'll evolve. So how you ask? Time, that's how. Trial and error. The risk takers will lead the way.

To be honest, I do believe many people are wasting too much money on huge data warehouses. I firmly believe we can get the data we need to act on without going so crazy. We're all too impressed with size. I mean, maybe if you're trying to clone Dolly or are the NSA or something, you can't avoid all that data...But size doesn't matter. Results do.

For example, a 5 terabyte data set of Twitter messages from random people no one cares about talking about what they ate for breakfast? Useless.

I know, it's unfair...We've spent millions of dollars learning about breakfast foods from Twitter and Tony the Tiger's family is still starving. Yup, it happens.

Just give it time. Big data and data mining on the web is still in its infancy in terms of application.

I am not at all convinced that our ability to convert data to actionable decision making metrics is limited.

In fact, I do just that every day.

There are definitely limiting factors, but the ability to make use of the data is much stronger than people commonly believe.

For structured data you just need a good understanding of the data structures and their relationships. Unstructured data (tweets, statuses, etc.) is much more difficult but there are several technologies available and under development that tackle unstructured data specifically. This even includes some high end tools that monitor voice data.

If you have a specific data scenario in mind I can be more specific. If you'd like to arrange a call I'd be happy to provide more detail.