Jason makes a great points there, but the speed issues could also be related to the very structure of your site. Many modern Wordpress themes rely heavily on JavaScript to load things like sliders, widgets and even page content.

You should use a well-supported, modern theme that is built for speed.

Page caching can make a huge difference, but in a lot of cases there are 2-3 page items that should be made static content rather than dynamic JavaScript content.