I've been a developer, consultant, manager. I've conducted hundreds of phone screens and interviews. Even now, I can tell you that it's difficult to gauge someone's experience, particularly in a short timeframe. I doubt I could accurately gauge someone's experience in three questions.
Technical assessment is hard. There's no simple solution, so I can't give you something totally reliable. The two most reliable ways I know to assess someone are:
- Referral: If you trust someone's judgement and they've worked with that person before and they strongly believe that person can help you, that's usually been pretty reliable for me.
- Actual Work: If you have a very small list of candidates, preferably one that you're seriously considering, start doing some actual work with that person, but keep your options open. Start building trust with regular delivery of working software. Get some early deliverables defined, have that person deliver you an early work product -- maybe a prototype. Continue to build on that incrementally. Either the candidate will show you with each incremental delivery that they can get the job done, or they won't, and you'll be able to make ongoing decisions about whether or not you want to trust the rest of your project to that candidate.
But, back to the meat of your real question -- how would you quickly assess a potential candidate to see if they might be a good fit? To be honest, I think trying to assess detailed technical skills if you don't have them isn't really going to work. Any question I could give you, they could fake easily enough if they're more technical than you are. You'd be better off trying to relate to them in the way that you expect to -- as someone who doesn't know programming and will rely on them to bring that skill-set. Ask them questions you want answers to about how easy and hard parts of your project will be. Ask them to help you understand how they will accomplish parts of the project. You won't necessarily be able to assess the detailed technology they're describing, but you will get a sense for whether or not they seem like they know how to go about your work, and whether they'll be able to communicate well with you and give you the information you need.
Once you find the right person who can communicate with you, then go back to my earlier response, and work on building trust through continuous delivery of software.
