Hey everybody,
I’m currently gathering examples of A/B testing from various websites, everything will be added in an Airtable, which will be embedded in a website. The plan is to have 33% of the DB available for free, and the rest under a paid plan, something like $14.9 per year or similar.
For now, I have the following information for every example:
- Company/Website
- Website Type (SaaS/eCommerce/Content etc.)
- Website Business Type (B2B/B2C/B2B2C)
- Website Industry (Web design/Development/Marketing/Automations/Finance etc.)
- Specific page/URL where the A/B test was live
- Variations (2/3/4)
- Winner Variations (if applicable)
- Date when the test was first detected (if applicable)
- Date when the winner variation was live (if applicable)
- Elements used in the test (Homepage banner/Checkout page/Product page/Pricing page buttons/Landing page bullets etc)
- Winning logic (a couple of sentences of why we think the winning element was better than the variations) (if applicable)
Please note that everything is still WIP, some of the above might be removed or we might add something new if we can.
Right now, I need help with a couple of questions:
- Is this something of interest? Even if you’re not willing to pay to have access to the rest of the DB, I would still be interested if you guys are interested in this or not.
- Besides the information listed above, is there something else you would like to see/need?
Looking forward to hearing every piece of advice you might have!
Tks
Yes definitely. As you know these elements are super important pertaining to to this information. There are plethora of small businesses that cannot the technical expertise needed in some cases to increase in engagements on websites, which for most websites are the best promotion tools. This would be a great asset for small to medium size business who want to increase the proficiency of engagements for one. Some other ideas I could think of too. I would explore those engagements with those users to see if there is an noticeable difference before and after using the information provided from your db.
Great idea - I'd be interested, especially in pricing A/B test results. Are you up and running already?
There are already a couple of products in market like this, goodui.org being the main one.
However, personally, I don't believe that this kind of meta-analysis of experiments from different sites has too much value, for a couple of reasons. I'm only really outlining this stuff for you because it might be food for thought if you want to create products for the experimentation community:
Whilst the theory of meta-analysis is technically sound, it is also fraught with issues relating to the complexity of web experiences:
Tests must be subjectively categorized and then analysed as 'patterns' when there is zero evidence that that thing was the cause of the result. For example, you might think a test won because you removed the voucher code box, but it equally could have won because of the change in page layout.
No two tests are in any way the same in any controlled way. Meta analysis mainly comes from sciences where experiments can be repeated under (relatively) identical controlled conditions. This is impossible and A/B testing cannot really be compared to that kind of controlled experimentation. By comparing two different experiments you instantly remove any control you had around audiences etc because they will not be the same.
The greatest benefit to be had from experimentation is by integrating it with business and brand strategy. A good business strategy is aimed at carving out a unique and differentiated direction of travel for that brand. The opportunity with experimentation is to iteratively develop and innovate the business through learning and adaptation. This means that experimentation should aim at being meaningful within the context of the brand strategy. Meta-analysis of experiments from different sites is, in many ways, the opposite of this because it seeks to make the ‘lowest common denominator’ changes which would be beneficial to any business anywhere. Conversion uplift and revenue might be achieved but what has been furthered in terms of strategy and differentiation? This is not to say that it isn’t valuable if you simply want to do ‘CRO’ but my argument is that nobody should merely do CRO.
