What's your opinion on using something like usertesting.com vs. real time usability testing (online and offline)?

UserTesting can be instructive in terms of understanding whether people understand your copy, CTAs, and intended flows but generally, I've found the quality of their panels to be pretty low. You're almost always getting people who are not your actual users, so the feedback can only be generally applied as above. I find whatever web analytics package or packages you're using are generally able to provide much better insights.

I also really do believe in *real* user panels. Buying pizza or offering small financial incentives to real users to click through new flows where they are talking out loud or answering specific questions is going to give far more actionable insights than anything else.

What I like to do is take my best guesses as to what's not working or what I'm looking to improve and then discover/validate via real in-person customer panels.

Happy to talk through this in more detail with you in a call.

I do both. We setup usertesting.com every Friday to run through a preset script for both our web & mobile apps and then review the videos as a team and add a couple items to our backlog. The users are definitely not our ideal customers, but their perspective can be quite enlightening and it's just a great thing to have automated and run for $80/week.

As for user tests, we actually use Clarity to find target customers (not hard for us) and then schedule calls and use join.me to either show them clickeable prototypes (made with Keynote -> saved to PDF) or watch them interact with our app and ask questions. We typically only request calls with people who are $1/min, and most calls last 20 minutes - so it's pretty cheap testing :)

Hope that helps.

P.S. I still do manual / in person testing with anyone who's a target customer and has a laptop or iphone - but that's just my personality :)

Usabilitytesting.com recruits users from a pre-selected panel of users who have given consent to be a part of usability tests in general. When you test with these set of users, what you hope to catch are low-lying usability fruits like fonts, colors, labels, widgets, etc.

Key issues like navigation and intended behaviour nuances will not get addressed as the users are not 'intrinsically motivated' to complete the tasks. The users are neither motivated to apply for a Certificate of Deposit online nor interested in buying a pair of hiking shoes online for the upcoming K2 climb.

Usabilitytesing.com is great for things which most people regularly do online - buy pizza, check account balance, etc. However, it is not good to conduct in-lab usability testing when specialized people with intrinsic motivation for completing a task are required.

There is benefits and drawbacks to every usability testing tool. People like usertesting.com because they can "set it and forget it" meaning they recruit all of the participants for you and offer automated unmoderated tests, then send you the results. It's an easy way to get a lot of feedback quickly.

While this may seem like a great way to save time and money, how does it really get you closer to achieving your goals of better understanding your users? Do you think that a test conducted in this manner is really attaining high quality results and valid data?

Empathy cannot be outsourced. It requires time and effort to get to know people at an intimate level. If you have stakeholders demanding usability testing with 100 participants and you're considering using a tool like this to save time, first consider educating your stakeholders on why it is unnecessary to conduct usability testing with that many participants. Five in-person test sessions will cost a heck of a lot less, take even less time, and result in very high quality and valuable feedback you can implement immediately. Plus it will help you and your team better internalize the flaws of your design and the needs of your users.

If you have more questions about these tools, usability testing, or other research methods, please feel free to call me on Clarity.