The best way is to lean how to ask great questions and to physically spend time with your users watching them work.
I don't have all the killer questions, but my favourite to help understand product roadmap opportunities is "What do you do 3 minutes before, and after you use our product?". That helps me understand where the opportunity lies to build a solid solution end-to-end.
Also, finding a set of users that feel comfortable having you hangout and work with them, taking notes and asking questions is invaluable. Just to understand their environment, the way they organize their work, and how they might communicate around the problems your solution is/could help with is incredible.
Every time I'm at an event and someone ask me what I do, I always ask them if I can "show them" and if they say yes, I have them find / download / signup and explore our app. Then I ask them questions about how they might see someone using it, who they think might be the best customer, etc.
It's not about not knowing, it's about continuously learning and testing different perspectives.
As for tools I use:
www.usertesting.com / especially new mobile tests!
I love what Dan said about asking what users do 3 mins before and after using your product.
I think the best product people know how to extrapolate better than others. Its one thing to gather feedback and observe users, but its much more challenging deciding what to do with that information. Simply asking users what they want wont always give you the best outcome. Someone once said, although Henry Ford is often credited, "if i had asked, people would have wanted faster horses". Theres some truth there. Its about the ability to observe customers and extrapolate the features.
Trying to understand what job your product is being hired by the user to do is another framework to uncover users unstated desires.
Observing your user while they are using your product while going about their daily routine with the above question in mind is one way to get to this.
Definitely spend time with your users. I would observe them doing their daily tasks (around the product you intend to develop). I would first just observe and then later follow up with questions on a second round of observations. You are going to want to understand their goals, their hopes, their fears, their frustrations, and their moments of using their tools before their process, during their process and after their process. Usually those unspoken moments will give you a lot insight.
The hard part is asking those questions in the moment and how get what you need answers by those potential users. A UX expert will understand how to pull out the information you need (which I train both UX employees and Product owners in gathering insights). I would be happy to answers any of your questions in this research gathering stage and what to do with that information going forward.
This is something which makes or breaks the entire product road-map. If anyone cracks this to a good extent, they will have a solid, sustainable product. However what many startups fail to understand is that it is about the experience now and not just the desired out put from the product.
As you have rightly asked in your question (beyond product), I would strongly recommend creating a mind map of your end user and highlighting the areas which will trigger the user to use your product and then the areas which will just add sweetness to the usage. Once you create a mind map, test it with different user profiles to build it more robust and clear.
Essentially, there should two or max three steps prior to the actual usage which will trigger a user to use the product and similarly two or max three steps post usage which will only sweeten the deal for him and make him stick for long.
