Just wondering how to run a closed-beta for an ios app.
1. How many users?
2. What kind of tools? (TestFlight? - recommend me!)
3. Other things to keep in mind
Apple will allow a developer to register 100 UDID devices per 12 month cycle to test via TestFlight or HockeyApp. Having started with TestFlight, I would really encourage you NOT to use it, and go directly to HockeyApp. HockeyApp is a much better product. There is also enterprise distribution which allows you far more UDID's but whether you qualify for enterprise distribution is difficult to say.
As part of your testing, I'd encourage to explicitly ask your testers to only register one device. One of the things we experienced was some testers registering 3 devices but only used one, essentially wasting those UDID's where we could have given to other testers.
Who you invite to be a tester should be selective as well. I think you should have no more than 10 non-user users. These people should be people who have either built successful mobile apps or who are just such huge consumers of similar mobile apps to what you're building, that they can give you great product feedback even though they aren't your user. Specifically, they can help point out non obvious UI problems and better ways to implement particular features.
The rest of your users should be highly qualified as actually wanting what you're building. If they can't articulate why they should be the first to use what you're building, they are likely the wrong tester. The more you can do to make them "beg" to be a tester, the higher the sign that the feedback you're getting from them can be considered "high-signal."
In a limited beta test, you're really looking to understand the biggest UX pain-points. For example, are people not registering and providing you the additional permissions you are requiring? Are they not completing an action that could trigger virality? How far are they getting in their first user session? How much time are they spending per user session?
Obviously, you'll be doing your fair share of bug squashing, but the core of it is around improving the core flows to minimize friction as much as possible.
Lastly, keep in mind that even with highly motivated users, their attention spans and patience for early builds is limited, so make sure that each of your builds really make significant improvements.
Happy to talk through any of this and more about mobile app testing.
We really benefited from allowing ourselves ample time to test the app. Firstly, it helps you perfect the app but we found more importantly, it actually helped with the marketing.
We used TestFlight but HockeyApp is also great. (neither solution is a joy to use, mainly due to limitations Apple places on distribution of test builds). Getting 10-50 users to test is usually sufficient unless you're testing a multi-player game that requires more players to test the in-game dynamics. In those cases, it's often better to go straight to a soft-launch in a smaller English-speaking country (see http://blog.tapstream.com/post/71538606229/the-art-of-soft-launching )
I mentioned how your beta test can actually help your marketing efforts: use the beta to create buzz and aura of exclusivity around your app. Get people with large social followings to test it, enjoy it and talk about it before it's released. Have a LaunchRock or similar lander to capitalize on that buzz and to collect emails for a waiting list to use for launch. This can give you a serious ranking boost on the launch day!
Keep in mind that regardless of the tool you choose you will need to setup each device and register it on iTunes Connect. Note also that there are limits:
* 100 users/devices per Developer Apple ID (Apple seems to me migrating to a 200 devices limit)
* Devices can be removed/replaced just once each membership year
* A time limit of 90 days
There are some alternatives:
* Ad-hoc installation (needs physical access to the device)
* Testflight (https://testflightapp.com/) the one I use and recommend +++
* Pieceable (https://www.pieceable.com/) lets you test over the web browser
* AppHance (http://www.utest.com/apphance) is a TestFlight like service
* HockeyKit (https://github.com/TheRealKerni/HockeyKit) is an opensource stack to host beta apps on the web (still needs the device registration on iTunes Connect)
* HockeyApp (http://hockeyapp.net/features/) an TestFlight like service
Go for fabric.io by Twitter, simple delivery, crash analytics etc... Way better than the iOS Beta program
The other solution is to get an iOS Developer Enterprise program (200$/y), you'll have unlimited devices.
In any cases, I recommend fabric.io for crash analytics, and high level user engagement stats
There is a tool called Daiwa that a number of developers use for beta testing the IOS apps. you have to share your UUID of your device and then they can send you an install link.
You can also get added as a tester in Itunes and pre test you app from there. This takes a little more effort to setup and get it to work.
Another tool I found that will let you install apps without iTunes is FunBox.