What are some solid ways for a SaaS startup to narrow down the market and estimate its size and the opportunity for acquiring paying customers?

Looks like you have a problem.

Based on what you've provided, you made the SaaS before confirming there were buyers for it.

Biggest mistake I see in SaaS. Second is trying to sell by Demo (features presentation) and believing that's going to work.

Better find some buyers.

I'm not seeing a demographic. Industry, size of company, department the SaaS best helps, who the stakeholders are that are involved in the buying decision, budget, problem(s) solved by the SaaS and so on.

Those are the things you'll be able to use to estimate the market size.

Right now you need customers. What pain points do they admit to having that your SaaS fixes?

When you get some customers, what are the commonalities? Do you have some already that you can look at?

Then you can tune your marketing to point at the type of companies, people and problems buyers admit to having.

That's the key, though: they must admit to having the problem.

If you don't have that, if you don't know why people would buy your SaaS, you're in real trouble. And I'm not talking about what YOU think. I don't know how many times I've gotten involved with a market, or started in on behalf of a client, and discovered the issues I thought would be important meant little to real buyers...and the things they valued were a stark surprise to me.

If you want help with this process, you know where to find me ;-)

Build a PowerPoint presentation that describes a) your company and vision b) the problem your product solves or the opportunity it creates for the customer, c) a description of your product including pictures of a non-functional prototype d) the fee you plan to charge and additional / optional services you will provide and finally e) summarize the benefits / ROI for the customer. Should be about 20 minutes or so. When you schedule introductory meetings, leave an additional 20 minutes after your presentation for feedback and discussion. Present your deck to a few prospects, refining/improving after each meeting. After a few meetings, you'll have a much better idea of the size of your opportunity and, equally important, launch timing and/or sales cycle. Instead of a beta followed by X% of the market will yield Y revenue, build a two or three year launch plan that says you will add X customers the 1st month, Y the 2nd, etc. That will give you a better understanding of cash flow and the timing for adding product development, sales and marketing resources. Good luck. I'm available if you need help.

Jason has explained the situation and how to change it to your favour - I agree with his assessment.

Let me elaborate a little and add some specific exercises you can perform.

As per the previous answer: you have to sell something before you build it. You can still - in a roundabout way - go back and do that. Set up meetings (real or virtual) with five top customers (current or prospected) and let them do all the talking. Ask about the current solutions they have in place, what they like about them, what is lacking, what changes often for them, what other parts of their business can affect that part that your product touches.

In addition: 2 exercises for you.
1 - Assuming you are B2B: Understand the buying hierarchy. There is your client, the buyer (the individual using the product) and the end client (the person it serves). This might be 3 separate people, or 2 or 1 but you must understand those roles. More here:
http://rgbrizi.com/revenue-development/the-three-clients-of-b2b-sales/

2) Understand your buyers' needs from three perspectives: Technical (how they want to save time and effort), Business (how they want to impact the business bottom line), and Personal (how they want to improve their own lives). More here:
http://rgbrizi.com/revenue-development/understand-each-client/

This should help get things moving.
If you want to discuss further, or assistance in these exercises, I am available.