What are some signs my startup may be "too niche" to ever become mainstream?

Niche is a good thing! It'll help you stand out. I would do a lot of cold emailing and targeting to that specific niche to dominate the market.

You can use something like http://www.growthok.com to help you find targeted contact information. Hope that helps!

In my experience every product or feature finds traction with 1%-5% of users - indefinite retention, express strong satisfaction in surveys etc. For paid products the number is usually much lower. The questions you must answer are:
1. What's special about these people? Don't assume they are "techies" or "early adopters" - these answers are too generic and often wrong. You need to reach out to your best users and interview them. If you do enough of these a profile will start emerging - single parents living in suburbs, <50 person companies in the transportation sector ...
2. Why do they love it? What job is it doing for them?

Once you have the answers to both you'd be in a good place to answer the question you're asking - is this a seed of a much larger market that's will grow significantly over the next few years.

The better question is: do you want to go mainstream?

If your startup is satisfying a niche, then your strategy might be to jump to other niches. If you're solving the needs of grocery shops with a small ERP, then you might jump to bodegas, to bakeries, to small coffee shops.

Going "mainstream" is totally different - it's the final step, after you've conquered all the niches that you can think of, where the product is "generic" enough to spread to dozens or hundreds of tangential markets. Most niche startups never even make this transition.

The real question, therefore, is not if you should go mainstream. It's: is your niche market (and the subsequent niches you'll conquer) big enough and will it grow over time? it's that simple. There's no worry about building a product only for grocery shops if they pay enough and there is enough of them. If you're not making money off of them as the market isn't big enough, trying to generalize features and "phase-shift" to the mainstream will not only not work but make you lose focus and turn your product to crap.

You can make all kinds of market estimations, but at the end of the day, has the niche market (or markets) paid you enough to survive, and, even if not, do you truly feel that you can capture more value from them (by adding new features and cool stuff) in the near future? If yes, viable. If not, run away.

Take "mainstream" out of your dictionary and mind and focus on niche market size and growth.

You can always scale up because you can always find some very related industry that can use your products/services with a minimal change.

For example, if you make specialized software for orthodontists and have captured the market, but have happy customers. Then, you should be able to add other types of dentists, maybe by creating a slightly different product that allows for their expanded types of services.