For the design? The downside is that you could end up with a very generic looking application/site. It's also not your (or your developer's) code so there will usually be a period of time required to get used to it in order to make any adjustments.

In the end, it may not really save you that much time...Depending on what it does or doesn't do.

Templates certainly aren't a bad place to start though. They can save time in many cases. Just not all cases.