SaaS platform with 10k niche users and a large company wants to buy a custom version. How do you model a cost structure beyond hourly?

You are the best judge of the cost and time estimate to do this. But as long as it does not take away a lot of time and attention from running your core business and growing it do it.

They way I would price it:

1. Hourly fees upfront or to be paid with milestones.(so that you are not worrying about cash flow problems or diverting cash to this while you could be using it for growing your core business)
+
2. Charge per user on top of the customization. Discount it but charge something. Figure out if they are charging for it from whoever is going to be using it, if yes establish a revenue share, very simply if they are doing all the work to sell and market it and you are doing all the work to build and maintain it then you should get at least 30% of the per user fee they are charging on top of the modification hours. You can also forego your upfront modification for getting 90% of the per user fees, if you truly believe they will do a good job spreading it around in the marketplace (get a commitment from them in writing of x number of users per months on a 12-24 month contract that they need to pay for)

2. If they are not charging their users, and giving it for free, figure out what value does it and to their core business. If it does add value to their core business then they should be willing to pay a price for it in addition to the modification charges.

Major impact of branching the platform - You will have to bug fix, support and make sure it's up. It can become a animal of it's own so this whole thing has to be profitable enough for you to support the dedicated resources you'll need. support it constantly and make you money in the process.

Also, look at co branding options, if possible. Don;t white label it if you can avoid it. It becomes a lot more interesting then.

Determine how much the custom modification is going to cost you in time and cost. Is this a fast track project? so something you have already considered doing but has less of a priority on your current roadmap.
If this is something that is on your roadmap you may what to not charge as much for the fast track. Of course you need to determine this.
If one "lighthouse" company wants your system with some mods then why wouldn't "lighthouse2"? Expanding your market is a good thing.
At the same time how much is this project/mod going to affect your focus on your system and roadmap? Can you afford to delay your roadmap?

As for branching, you need to figure out a way to add this Mod as a module allowing you to easily integrate. You should be building your system in a modular manner anyways. Turn off/off features easily based on customer wants/needs.

Figure out a way to do the mod for the light house customer. :)