Is GitHub the right solution for startups to host private code?

Simple question: Simple answer. Yes.

It's a great code repo. As a non-tech founder, it's still usable and doesn't take a lot to use to its full benefit.

Yes, it is the right solution, but you should also consider Bit Bucket, as they offer unlimited private code repositories, which might be less expensive than GitHub.

Most likely yes. GitHub is the de-facto standard for git hosting (public or private). There are very good competitors like BitBucket. But unless you can justify why you need to use a different provider, GitHub is a good choice because everyone you work with will already know how to use it.