I think you have answered your own question:..."it could have a negative effect". So let's assess the risk.

As James Goldsmith said: "if you pay peanuts, you get monkeys". This is particularly true in sales.
Let's just follow this logic for a moment. If you pay an agency $4/hour, they probably keep $2 and are willing to part with $2. Assuming the telemarketer gets paid $2 an hour; $16 a day MAX; they HAVE to come from the deprived countries. Is there English good enough? Are they aware of local culture? Are they going to stay motivated?

This is very likely to have a negative effect on your brand, which is never good. No shortcuts I'm afraid.

Good luck though.