You can quit from anything for any time for any reason. You, or other people reading who want to leave a similar arrangement, may have an operating agreement in place which describes what happens. You still have shares to deal with, for instance.
Next time, work out a way of dealing with people up front. That way, disputes will have a path to resolution instead of ending up in this emotional mess you're in.
The two of you sound like neither of you really understands business or how business ownership works. Is there value in this business? If there is, you should consider talking to an attorney to protect your rights and property.
Your co-founder's concern that nobody will fund the idea if you leave is less important than he believes. People make changes all the time. The question is: does the idea have life in it? If there's genuine value, your being there or not won't have much impact on funding decisions. Yes, founder credibility is important. But it's not the only thing, or the main thing.
You are way too wrapped up in this thing emotionally and my advice is that you find ways to disconnect yourself from it entirely (including ownership) ASAP.