A conversion specialist is the person who looks at your landing page and knows which headline, which order, which button will make more people buy. Good ones are rare and they are expensive, because what they do compounds.
A few points of conversion is the difference between a marketing budget that pays for itself and one that does not.
If you are a large company you hire one. If you are a four-person marketing team, you do not, and your page converts at whatever it converts at. Forever. Nobody calls this a problem. It is just the shape of the industry.
We decided to build the expert instead.
Not a tool that gives you advice about your page. A machine that runs the experiment continuously, learns which version works for which visitor, and sends each person to the one most likely to convert them.
The specialist's judgement, as a running system, for people who were never going to be able to hire the specialist.
Then we had to answer the obvious question, and it is not a question you can answer with a chart.
Is it actually as good as a person?
We could have published a benchmark. Everybody publishes benchmarks and nobody believes them, including the people who publish them.
So we put it on a stage, at a conference, in front of the industry.
A live head-to-head. Human conversion specialists on one side doing what they do. The machine on the other. Same pages, same traffic, same question.
It won.
In front of a room of people who do this for a living, and who had every reason to want the other result. Nobody wrote the margin down, so I am not going to put a number on it.
And then the part I actually cared about.
Beating the experts was never the point. It was the only way to make anyone believe the point.
The specialists were never our customers. The people who could not afford a specialist were.
In the alpha, ordinary marketing teams, the four-person kind with no conversion hire and no plan to make one, were seeing 21% more conversions. Not against an expert. Against the page they had before, which was the only alternative they actually had.
That 21% is two things, and it would be easy to sell it as one. The allocation is the visible half. The other half is that we moved the statistics to a Bayesian model, so a result reached significance on a fraction of the traffic.
Most of our customers never had enough traffic to finish a test. The machine was only useful because the maths underneath it stopped needing so much of them.
That is the whole product. A capability that used to be a salary became a feature, and the people it reached were the ones who had been quietly priced out of it. Two patents came out of the work. The patents are not the interesting part.
The demonstration was a design decision, not a marketing one.
We could have shipped it with a number in a blog post. The reason to do it live, against people who did not want to lose, is that a claim you can watch being tested is a different kind of claim. The audience did not have to trust us. They watched.
I think about that more than I think about the result. Most product arguments fail because they ask to be believed. The ones that work let somebody check.