Based on my experience, the biggest challenge is getting the right talent that can pull off a quality work, I have done this by sending out small tasks to different developers and evaluate their works after. (But I guess this isn't your issue since you have 4 yours experience of working together). The next challenge I faced was time difference. Both parties need to be flexible enough with their schedule meaning working in the evening to get things resolved right away. Exchanging email with hours/one-day-delay to reply kills the momentum and motivations.

Let me know if you would love to get more information about my experience with outsourcing.