Loading...
Answers
MenuWhat is a sufficient number of people to do A/B testing with? 10, 20, 100, 1000? Why? Should this be a closed group initially or not?
This question has no further details.
Answers
There's no set right answer. The more people the better the data. Traditionally most sample sizes are 100 people but think about it like this. Which has more power? 90% of 100 or 90% of 1000 all vote in your favor. Which would you pick? My recommendation would be use a sample of at least 100 people. 👊🏽
Simple Answer - 1000. However, dump a losing ad quickly. For example, if ad A is pulling in a 5 to 1 response over ad B, come up with a new ad B that attempts to beat ad A. You may see these results almost immediately. Examine the winning ad and beat it. Do not get hung up on the sample size if an ad stinks.
I hope this helps.
Best of Luck,
Mike
PS I may assist you as my time permits. Right now I am doing something that I haven't done for many years. I am actively soliciting new select clients due to selling one of my businesses and freeing myself up a little. But as you might expect, I am still very busy. That is how I like it.
TheLittleGuysWayToRiches.com
Statistically speaking, a pool of 30 respondents is enough to start seeing a trend. 100 is about the maximum. Beyond that it's just more of the same pattern and it might even start getting confusing, because the differences will start to level out.
Related Questions
-
How do you get access to the right people to validate an idea?
The art of the "cold pitch" is definitely something that needs to be worked on, and doesn't come naturally to everyone. A couple of quick tips: 1. 4 is not a big enough target group, you've got to cast a bigger net. Try pitching 20, and aim to get 3-5 responses. 2. When sending a cold email, really think about what you are offering them. Whilst you'll get the odd good egg that simply wants to help - you can't expect entrepreneurs and small biz owners to take time out of their day to answer your questions. To counter, why not tell them you're conducting research in the space, and would be happy to send the finds/reports back to them in a nice format, which could in turn help their business. 3. Don't give up, keep hustling. Try changing around the emails slightly, track which emails convert into responses, and fine tune from there. Good luck and feel free to book a call if you want to chat more. Best James.JP
-
Other than appdata.com, where can I find download & revenue numbers of successful iOS apps?
I believe Appannie.com is what you are looking for.AC
-
Is there a way to use paid advertising to test a domain name?
Yes. Use the "Link Text" in your facebook ads to A/B test various domain names. To do this, simply launch a few identical ads, and only change the link text to say the variation of the domain name you are considering. The ad with the highest CTR (not conversion rate or other KPI) will be the one that sparks the most interest in your target audience. Note: one domain may have a high relevance to one audience, but low relevance with another. So this tactic should only be used if you have a very specific demo you are targeting.AG
-
My direct competitor is growing fast, my start-up is still at an early stage. Do I need to cancel the initiative and try something new?
Even if your competitor continuously capitalizes on your ideas, by imitating your product features, etc., and even if he grows exponentially faster than you, it shouldn't intimidate you. Similar product, less money, whatever else, doesn't matter. Get awesomely creative with your marketing. Focus on the user experience and forget the competition. That's one thing no competitor can ever take away from you. And, the only thing you've got to mediate risk. I'd love to share more with you over a 15 minute call. I can guide you through getting a better understanding of your customers, and developing an awesome user experience, by moving beyond conventional marketing. Looking forward to it. AdrianAC
-
How do I do the right Market Research for a Product or App?
The best way to do validation on this idea would be to acquire customers through Facebook ads (rich targeting makes you able to cohort users nicely). Then, you could manually pair people via email or through a simple text messaging exchange facilitated by Twilio. In this scenario, you could prompt them to take the actions you think will be at the core of your app's functionality. Observing their engagement rate and what they do with each other would give you some insight into how viable this is. Happy to talk this through in a call with you. I've helped a lot of clarity members through similar customer development exercises.TW
the startups.com platform
Copyright © 2025 Startups.com. All rights reserved.