Typically open-source projects that end up generating a profit have an active product that's already being used.

So step 1. Is to open source it and see if anyone uses it - then you can make money in a bunch of ways.

Checkout Wordpress, MySQL, Linux for examples - but typically it's providing paid devices for either hosting, managing or customizing the software.

The hardest part is getting others to use and contribute to it.