I work for a web design agency. We are trying to make decisions based on data. How do I know which user tests to run and when?

Sorry to be a bit blunt in my answer, but the client hired your "e-commerce web agency" to do e-commerce conversion optimization, yet your company doesn't know how to do e-commerce conversion optimization?

Perhaps have a conversation with someone in your company or talk to your manager or the agency principal and encourage them to hire for e-commerce conversion optimization expertise. It's an entire discipline requiring years of experience and track record in driving higher conversions for e-commerce sites (in my opinion).

In lieu of that, how many marketing dollars are going to be spent to drive traffic to the site and who is managing that part? Often new start-ups will spend so much of their dollars BUILDING the site, they don't leave much money to spend on AdWords or other methods to drive traffic to it. The amount of traffic will help determine how many tests to run and how often. Usually we start with two very different landing pages (do you have landing pages?), then fine-tune the split testing down to things like headlines and photos. The checkout pages, especially any pricing pages, are also critical in the testing. Is it a brand new concept or established product/service? If brand new, it will take very well-crafted pages to get visitors to pay money for something (vs. a trial). If established, why is your product best and why should I buy now? You should map out the conversion flows to the confirmation page (success page) and measure the drop-off or conversion rates at each step. Test different elements/pages at each step. Also measure things like average cart size (if more than one product), revenue per transaction and long-term customer value.

Hope that helps. Best wishes to you!

Depending if the site has already been built and is going through a redesign VS building a brand new site is very different processes.

Joe points out some critical starting point questions thaty you should have clear answers for in order for you to design with the user in mind.

SITE DESIGN
Much of what we're talking about is building UI design from usability and information architecture methodologies to create the site.

SITE PERFORMANCE OPTIMIZATION
The next step to optimization is to look at site performance and determining if there are areas of improvement which can improve your funnel or optimize your sales and test for varying designs (based on your hypothesis) and use analytics to answer your questions.

MARKETING OPTIMIZATION
Lastly, it's looking into the marketing optimizations where you evaluate marketing ROI, testing of landing pages, and evaluate various marketing channels.

Experts above all offer great information on what to pay attention to. But as mentioned, without details, we are all just making assumptions.

Feel free to schedule a call to discuss further.