Using LLMs to create A/B test ideas, for conversion rate optimization, is becoming quite popular.
Naturally we have the following concerns
1. How do I know the test ideas are tailored for my business?
2. Is an LLM confined to best practices, or do they have the ability to understand business context and create unique wins?
To answer these questions we’ll do a case study of an e-commerce store selling gifts for loved ones. The hypothesis is
If we include social proof, in the form of a badge for number of visitors that have previously purchased the product, we'd expect to see higher conversion.
The test was negative. For the business, visitors purchase gifts hoping they're unique and special. We'll attempt to recreate this scenario with LLMs and detect negative outcomes early. Case study is taken from Samuel Hess live website analysis podcast.
Self reflection with business context
Let's see if we can recreate the social proof idea using LLMs.
Prompt Create test ideas for an e-commerce store selling unique gifts online. Expand on social proof ideas.
Response
...
5. Trust Badges and Certifications
Test: Display trust badges (e.g., "Trusted by 10,000+ Customers", "100% Satisfaction Guaranteed") near the product description.
Objective: To enhance credibility and reassure users of the reliability and popularity of your store.
Great, we recreated the idea. Now we'll utilize self reflection capabilities of LLMs. In the process, we'll include business context which could have been incorporated through the business owner, customer surveys, or by checking initial assumptions.
Prompt I have a gift shop e-commerce store. The customers use it to buy special gifts for their loved ones. Ideally, the gift bought should seem novel for their loves one. Should there be social proof for how many people have bought some of the items in the past?
Response
Given your goal of making the gifts seem novel and rare, displaying social proof with high purchase numbers could contradict that perception. Instead, you could emphasize the exclusivity of the items by highlighting limited availability, unique craftsmanship, or personalized options. If you still want to include social proof, you might consider subtle approaches, such as showcasing testimonials or stories about the unique experiences people had when giving the gift, rather than focusing on the number of purchases. This keeps the feeling of rarity while still building trust and credibility.
It's clear that some level of reflection to weigh ideas can be beneficial, even for widely excepted ones like Social Proof. It's commonly understood that LLMs only produce "best practice" test ideas. But its likely that "best practices" are not the issue, but rather the lack of proper business context.
Conclusion
LLMs seem to have enough reasoning capabilities to detect negative A/B test ideas - woohoo! It's context which is the bottleneck for both humans and LLMs alike.
Some examples of companies using AI for winning A/B test ideas are Sitewiz, Heatmap, Evolv AI, and more. Their challenges lie in obtaining, organizing, and consuming all possible information of the business and its customers effectively. That’s where unique engineering solutions arise and AI expertise takes over.