One of the best ways to identify real problems in a market is to be a consultant and actually get paid to help solve the problem. Many great product companies started off this way (ex: Freshbooks, Hootsuite, Shopify, etc). They solved their own problem, and then build a solution so that others could use it.
If you're already a startup, and you're trying to refine and learn, then you can do more hands-on deployments for customers, where you actually go in and help them solve the problem with your software. Now, if you start to get paid by the hour to do this work, then you went to far.
Either your a service company learning and getting paid, so that one day you can spin out a product, or your a venture backeable startup and doing things that don't scale initially to learn how the customer thinks.
The best question I've ever heard to truly understand a customer is to ask "What do you do 3 minutes before, and after, you use our product". That's where the opportunity lies.
A lot of it is being observant - and doing research. Many businesses are based on solving problems the founder had him/herself (e.g. "this seems busted - there has to be a better way") or things they recognize because they are in that line of work (e.g. "man, this could be some much easier if thus and so happened"). That said, many other very successful businesses start from identifying a huge market area and looking for problems worth solving - and seeing if they can be, profitably.
Then you put a smart team against it and iterate.
There's no one right answer.
That said, simply using self reference criteria isn't often the right way to go. I spend a lot of time talking with founders or founding teams who are recent grads trying to do things for that market because they are most familiar with it, even though it might not be the best or most profitable fit, for instance.
Better An Existing Problem
Many startups were not the first in their domains. Take Google for instance. Many popular search engines existed – Altavista, Yahoo, etc – and we never thought we would move away from all of them at once.
Adapt Existing Solutions To Problems In Your Country
Many ventures exist which essentially do the same thing, but with localization. For example, SnapDeal is a Groupon clone in India with localized content and features. Google’s clone Baidu is hugely successful in China. And Singapore has its very own Pinterest clone called Singterest.
Apply Existing Solutions To Your Industry
Think Linkedin. Social networking concept adapted to the business industry. Think Airbnb. A popular Hotel search and booking concept adapted to the Bed and Breakfast industry.
Seek And Address Existing Pain Points
Every business vertical has issues that will range from areas in marketing, customer acquisition, customer service, sales, operations and others. Chalk out the industries that you identify best with and would be passionate about working within. Then, speak to a wide range of professionals from that industry to understand what problems they are facing in their jobs. Write all the information down in an excel sheet and then begin to narrow down into the most common problems faced by people and choose the area that excites you enough (marketing, sales, operations, etc). There you go. You now have a list of problems that require a solution.
