Loading...
Answers
MenuWhat's your opinion on using something like usertesting.com vs. real time usability testing (online and offline)?
This question has no further details.
Answers
UserTesting can be instructive in terms of understanding whether people understand your copy, CTAs, and intended flows but generally, I've found the quality of their panels to be pretty low. You're almost always getting people who are not your actual users, so the feedback can only be generally applied as above. I find whatever web analytics package or packages you're using are generally able to provide much better insights.
I also really do believe in *real* user panels. Buying pizza or offering small financial incentives to real users to click through new flows where they are talking out loud or answering specific questions is going to give far more actionable insights than anything else.
What I like to do is take my best guesses as to what's not working or what I'm looking to improve and then discover/validate via real in-person customer panels.
Happy to talk through this in more detail with you in a call.
I do both. We setup usertesting.com every Friday to run through a preset script for both our web & mobile apps and then review the videos as a team and add a couple items to our backlog. The users are definitely not our ideal customers, but their perspective can be quite enlightening and it's just a great thing to have automated and run for $80/week.
As for user tests, we actually use Clarity to find target customers (not hard for us) and then schedule calls and use join.me to either show them clickeable prototypes (made with Keynote -> saved to PDF) or watch them interact with our app and ask questions. We typically only request calls with people who are $1/min, and most calls last 20 minutes - so it's pretty cheap testing :)
Hope that helps.
P.S. I still do manual / in person testing with anyone who's a target customer and has a laptop or iphone - but that's just my personality :)
Usabilitytesting.com recruits users from a pre-selected panel of users who have given consent to be a part of usability tests in general. When you test with these set of users, what you hope to catch are low-lying usability fruits like fonts, colors, labels, widgets, etc.
Key issues like navigation and intended behaviour nuances will not get addressed as the users are not 'intrinsically motivated' to complete the tasks. The users are neither motivated to apply for a Certificate of Deposit online nor interested in buying a pair of hiking shoes online for the upcoming K2 climb.
Usabilitytesing.com is great for things which most people regularly do online - buy pizza, check account balance, etc. However, it is not good to conduct in-lab usability testing when specialized people with intrinsic motivation for completing a task are required.
There is benefits and drawbacks to every usability testing tool. People like usertesting.com because they can "set it and forget it" meaning they recruit all of the participants for you and offer automated unmoderated tests, then send you the results. It's an easy way to get a lot of feedback quickly.
While this may seem like a great way to save time and money, how does it really get you closer to achieving your goals of better understanding your users? Do you think that a test conducted in this manner is really attaining high quality results and valid data?
Empathy cannot be outsourced. It requires time and effort to get to know people at an intimate level. If you have stakeholders demanding usability testing with 100 participants and you're considering using a tool like this to save time, first consider educating your stakeholders on why it is unnecessary to conduct usability testing with that many participants. Five in-person test sessions will cost a heck of a lot less, take even less time, and result in very high quality and valuable feedback you can implement immediately. Plus it will help you and your team better internalize the flaws of your design and the needs of your users.
If you have more questions about these tools, usability testing, or other research methods, please feel free to call me on Clarity.
Related Questions
-
When people sign up to my online marketplace, should I ask for their phone number in the application form so I can get feedback from them?
I have designed a few of these online marketplaces and I can tell you from experience that in this day and age, phone numbers are a deterrent on a signup form. I advise against it and agree more with the cons. However, that does not mean that you cannot get feedback from students. Form the past websites that I have built with student demographics, I have found that they like to interact with each other, so a better way to get feedback is to set up a forum in your online marketplace. Also, just adding a comment thread like blogs have is a very good way to get feedback, even more effective than phone numbers. Also, one reliable method that I found works is an incentive based feedback survey when students can earn pennies on the dollar to take surveys. If you go this route, you could offer 25 cents or $1 or even $5 to the first 20 or 100 students that fill out the survey answering all your questions that you would normally have asked over the phone. I hope my suggestions are helpful to you. I can always assist you in building your online marketplace. I have more suggestions for you as well. BruceBC
-
How do I run a closed beta test for my mobile application? Development will be finished in 3 weeks.
You should try to engage people using social networks, it is easier to spread than email. The conversion rate on emails are low but is still a valid tool for that. Send and email with a simple and objective message that will make people want to try. The best way to have feedback from users is to watch them use the app. You should put them on the hands of everybody that you can and without any instrucions and just watch, don´t even say that the app is yours. Try to do it a lot. If you want feedback from others, you can include the feedback form inside the app and suggest users to answer occasionally. I would also strongly recommend to use a tool as Flurry Analytics. Is the best way to get data from how is the use of the application. Pay attention to those data and be open to change your app a lot, you may need more features or cut some off to make it easier to use. If you need more help please contact me.BS
-
If you launch a product to a few different countries, does it make sense to do usability testing in all those different countries?
Perhaps. You have to consider how your product is used and how the users will sensorially interact with it. If you have a website, for example, you could probably assume similar US and UK usability -- but now consider layout for Semitic languages which are read right-to-left -- or consider ideographic languages where the ideographs could interact with other graphical elements in a way wholly irrelevant with the languages you're familiar with. If you have a physical product, the question is conceptually the same, but the actions are different. You have to consider how users in a given culture use and interact with products of that sort. Different cultural memes, ways of learning and expectations can effect usability. There are some simple ways to do these things if you have a small budget. You could easily go to craigslist and recruit 10-15 natives of a given country or region. Give them the product or have them use the website (for 60-90 minutes), video it (with their permission of course and for reasonable compensation), and you'll get a potentially useful indicator of usability. If you're P&G or Microsoft, you've got the budget to test against statistically valid sample sizes pretty much everywhere in the world. You don't have that, so you need to use some heuristics. Good luck to you. Should you have any questions, I'd be happy to discuss.SM
-
How should the dynamic between a ux designer and a developer who are working together look like?
It depends a lot of in the skill sets and experience of both people but in most cases the ux designer should be controlling the developer pretty heavily in order to make sure his ideas come through properly. The UX designer may just need to work on his approach so people don't feel bossed around and more like they are working together. In an ideal world, there would be a project manager who makes sure everything is communicated well and keeps the dynamic feeling great.JM
-
What's your advice for drilling down on use-case for our real time communications iPad app for co-viewing docs together?
Hmmm? This is a tough one because your product is so wonderfully horizontal, meaning it applies to a very broad set of cases. I'd slice and dice in two directions. First, I'd think about the general process of collaboration on a document, for example: a) drafting - are two people actually writing the words together? I personally don't like this. b) first read through together staying on the same sentence or page, c) editing together, etc. Second, I'd think about industries or verticals or contexts, such as in a Legal setting, in a Tech Documentation setting, in a coding or XP dual programming setting, in a social media post drafting setting, and so forth. Check out: https://medium.com/look-what-i-made/fd48c3b412bc Seems to be evocative of what you're doing, but targeted to one specific use case of "pair programming".BB
the startups.com platform
Copyright © 2025 Startups.com. All rights reserved.