Hi, I am an Israeli lawyer and therefore the following does not necessarily comply with the American legislation. In Israel, you don't need a signed contract in order to have an obligating agreement. Oral consent (in mist cases) is enough and when you have a proof in writhing (such as emails) it's even better.
What was the oral agreement between you two? Did you issue him shares? If you did and he is a partner he bears the risks as well as the advantages so it doesn't make sense to ask the money back. In addition, how did you write the money in the company's books? If it was not written as a loan you might have a better case claiming the money was never meant to be paid back.
As a practical advice, try to build a mechanism that will "free" you from him. Maybe pay in installments, decide you will pay back from revenues or as a percentage from a further investment etc. You are welcome to give me a call and we can think together of creative ideas to offer him from the business point of you rather than the legal one. Good luck!