Typically open-source projects that end up generating a profit have an active product that's already being used.
So step 1. Is to open source it and see if anyone uses it - then you can make money in a bunch of ways.
Checkout Wordpress, MySQL, Linux for examples - but typically it's providing paid devices for either hosting, managing or customizing the software.
The hardest part is getting others to use and contribute to it.
