We have our three core team members testing currently, 100 testflight signups, they break down as friends and family, cool early adopter tech people, massive enterprise customers in retail and hospital systems, and physician early adopters. How do we think about rolling out our testflight beta? Over what time frame should we be considering? We are pushing releases essentially weekly at this point with a tiny team.
More information needed, I think. What goals do you want to accomplish with this beta? In terms of process and timeframe for actually _rolling out_ the beta, I don't think there's anything special that needs to be done.
When it comes to gathering feedback from those user segments, however, it would make sense to target them.
First of all, I'd suggest you use Hockey instead of TestFlight. Although I'm sure the product will change now that Apple has acquired them, TestFlight is currently inferior to Hockey in so many ways, I could write an entire post on just that!
Ignore "cool early adopter tech people and friends and family" and decide which of the user segments of potentially real early adopters are most likely your customers. This is best accomplished by actually segmenting each group (into a separate email list) and surveying each to understand their current perceived degree of pain for the problem you're solving and also trying to assess their degree of motivation to experiment with your approach.
The product should be refined enough to fully function before testing with real customers. This is a point I can't stress enough. Customers act the same regardless of what stage of development you are in. This is to decide they will decide quickly (within minutes) as to whether they think that what you have on offer is worth their time. If you fail to impress the first time, you will have a difficult time getting them to reopen the app, regardless of what the build notes have on offer.
So if you're still a ways away from having a true MVP, you should count on early adopter tech people to give you generalized product feedback. Their feedback can't be relied upon to validate actual product/market fit but they can help you shape the MVP to get the core UX right and help squash bugs, etc.
Happy to talk through this in more detail in a call.
We just finished a small beta with 200 people on TestFlight, so I'd be happy to share my experience! I've run 5 TestFlights now totalling many hundreds of users so I'm somewhat qualified to help you out.
My main advice here would be to roll out to users as slow as you can afford to, but obviously there's a load of factors to consider, notably how much testing you've done so far.
The issue with large scale rollouts is twofold. Firstly, there's a massive hit to your servers all at once, which could cause availability issues and highlight any bugs. Secondly, the more people you get involved early on the more people will SEE those availability problems and bugs. If you control the rollout you'll be able to see how your systems react to the increased load and scale accordingly.
That goes for platforms too. What's powering your iOS app? Do you have a web app? Secondary services that add on to your main platform? If there's anything that it's possible to hold off on offering, you should think about it. It's best to start out with something that's really great and reliable - you don't get a second chance at making a first impression.
Most of this stems from the rather uncomfortable truth that people don't actually understand what betas consist of. You're probably hoping for 100 people that really want to help you improve your product, but what you may actually have is 100 people that really want early access to something that's already finished.
It's best to manage their expectations up front. Just be clear about the feedback you're expecting, the level of contact you'll have, your update schedule, how you'll be communicating with them, how often, etc.
There's other factors to consider here too like the reliability of TestFlight (Apple are usually great, but TestFlight has its own unique issues with sending out invites and invalidating them). You'll also need to think about the drain that handling feedback will have on your team's ability to iterate quickly. It sounds like you're doing a great job at the moment, so you need to consider how much impact it will have dealing with support requests for non-technical issues.
I'd love to help you more. Sounds like you're working on a really interesting product, and betas are right in my area of expertise. Let me know if you'd like to schedule a call.
