Spray and Pray Marketing Is Dead: The Case for CRO and Experimentation with Joe Johnston

Zach:
A lot of marketing teams launch content, they launch campaigns, they cross their fingers, and they hope for the best.

But we can be much more successful if we form hypotheses, we test, we experiment, and we try to optimize for conversions.

In this video, I’m talking to a good friend of mine, Joe Johnston, in Bristol, England.

He’s a great guy. I really enjoyed this conversation.

He is an expert in conversion optimization and experimentation.

I hope you get as much out of this conversation as I did.

Let’s get into it.

Joe, thanks so much for joining us, man.

I really appreciate you coming along.

I always enjoy our conversations, whether it’s listening to you give a guest lecture or just the chats that we have.

I really appreciate you coming to talk to students.

I know they’re going to get a lot out of it, getting to hear from somebody who does this every day.

To get us started, why don’t you just introduce yourself and tell us a little bit about your background?

Joe:
Yeah, for sure.

Thanks so much for talking to me today.

It’s a great opportunity.

My name’s Joe.

I work at a company called Launch.

We’re based in Bristol in the southwest of the UK.

We also have another office further south in a place called Exeter.

My role is to head up the conversion optimization and experimentation team at the agency.

We’re a performance marketing agency.

That’s predominantly paid media, working with companies like Google, Meta, Pinterest, and TikTok to advertise on behalf of companies.

But my team’s specialism is in optimizing that whole journey through testing and also learning.

We’ll do lots of research to try and understand how people think, what their expectations are, what their motivations are, and what their needs are, so we can better cater for those as they go through the journey.

It’s a small-ish company.

Thirty-five people for an agency isn’t too small, but we’re an independent agency.

That means we’re not necessarily attached to the big global networks where there can be quite a lot of institutionalization.

So we can be quick and nimble, and we’ve got really good leadership.

There’s a lot of innovation happening around AI at the moment, which is really interesting, and I’m sure we’ll talk a bit more about that later.

Zach:
I just picked up on something you said there.

I often hear you talk about conversion optimization, and I know you guys have moved away from just doing websites and you’re doing paid ads now, or helping companies with that.

There’s some nuance in what you just said around performance rather than conversion.

Is there a shift there that’s happened for you guys that’s worth touching on?

Joe:
Yeah, that’s a good point.

Performance marketing traditionally would be defined more as the kinds of marketing that you can measure, but specifically it has a very strong emphasis on measurable returns.

So that’s usually paid advertising right at the bottom of the funnel.

People usually think about search marketing and then also demand generation and marketing within social media channels with a really obvious response.

That’s performance marketing, broadly speaking.

But actually, the scope of it has been changing over the last few years.

There’s been more of a push toward full-funnel performance.

Seeing that whole journey as being not just about the bottom of the funnel, where you’re talking to people who are ready to convert, but actually awareness through consideration.

People building awareness, building demand.

That’s how we now broadly define performance marketing.

It’s really all stages of the journey, but with a strong emphasis on measurable returns.

Being able to account for or prove the effectiveness of what you’re spending and your media spend.

That’s where the performance comes in.

In terms of experimentation, historically experimentation has kind of been the best-kept secret of a lot of web teams.

Traditionally, you would be A/B testing on a landing page or on a homepage.

Pretty much everyone on the planet has been involved in an A/B test at some point in their lives.

They’re just probably not aware of it.

If you’ve been on Booking.com, if you’ve been on Airbnb, if you’ve been on Google, if you’ve been on Pinterest, all of these big brands are experimenting day in, day out.

If you’ve been on a website and seen a specific headline or a different kind of button color that you might normally see, then that might be a test that you’re involved in.

What they’re doing is trying to understand the impact of making these small, tiny changes, or sometimes bigger changes.

That would typically sit within the web team’s responsibility.

It’s hard to say why it started there, but it’s definitely becoming more of a culture within businesses.

It’s spreading beyond just the web teams into product teams, advertising and marketing teams, content and production.

All of those different teams can see the benefits of it.

Customer service too.

Because the mindset, the scientific mindset of only changing things which we know are working and why they’re working, it makes sense for that to be a philosophy that everyone buys into.

So we’re seeing it move away from just being a website methodology and applying the same methodology throughout the whole customer journey.

Testing within ads as well as testing on the website.

Zach:
Okay, cool.

I was going to ask you to give us a description of conversion optimization or conversion rate optimization, but you’ve kind of just jumped into it there, which is great.

You’re talking about experimentation, the scientific method, A/B testing, multivariate testing.

Is there anything that you want to add to that around what CRO is?

Joe:
Yeah.

Whenever I ask the question, “What do people think CRO is?” everyone seems to have a pretty good general understanding of it.

I remember asking students at previous lectures before, and they always seemed to have a pretty good grasp on the idea.

Effectively, what you’re trying to do is optimize the experience.

That’s usually a website, but it could be elsewhere.

It could be an app.

It could be in an advert.

You’re optimizing that experience for the users, but doing it through a systematic way so that you can measure the outcome.

That conversion rate and that conversion can be a purchase.

It could be someone filling in a form.

It could be someone watching a video all the way through to the end.

Whatever that conversion action is, you’re optimizing that experience so that we can measurably improve what we mean by that metric.

Most people have got a pretty good grasp of that.

Within the CRO and experimentation community, there’s been more of a push toward calling it experimentation, just because CRO can limit the scope of it.

It’s not just about conversions.

It’s about other things as well.

But I think most people still recognize CRO as the term, so it’s probably the most useful one to talk about.

I still refer to it as that because more people use that lexicon rather than experimentation.

Zach:
Old habits die hard, right?

You touched on the fact that most students and most people you talk to have a pretty good idea about what it is.

Is there a misconception about what it is that you come across often?

Are there any misconceptions about what CRO is?

Joe:
Maybe the main misconception I see, especially within chief marketing officers or people who are quite highly paid or high-ranking in businesses, is they see it as a potential one-off project.

They’ll say, “We’ll just do a CRO review of a website.”

They see it as a quick fix.

“Let’s just spend three months intensely doing a CRO project. That will fix our problems and our conversion rate, and then we’ll move on.”

It’s seen as a one-off.

But that is similar to search engine optimization.

That will never work.

It’s a consistent effort that you have to put in to see the rewards.

In fact, anything less than six months, you’re not going to see any kind of benefits from it because of how long it takes to get the program up and running, do the research, start testing, and start doing enough tests that you see some winners as well.

It’s often a volume game.

It’s a numbers game with doing tests because the win rate is so low.

A good win rate is about 30% in A/B testing and CRO.

So if you’re running one test per month in a year, maybe two or three of those, or perhaps four if you’re lucky, will win.

If you’re an impatient CMO and you’re spending money on a program that in the first three months hasn’t seen any winners, maybe you’re going to cancel that program.

But really it’s about having the right expectations going into it.

In terms of misconceptions, it’s probably that it’s a silver bullet or that it’s a one-off CRO review.

It should instead be seen as more of a program that you invest in and try to build a culture around.

Zach:
That’s so good.

People see it as a short-term thing.

That short-termism where we’re focused on a quick project, a quick win, then let’s move on to the next thing and spend that money elsewhere.

But it needs to be an ongoing testing process.

That’s awesome.

Let’s shift gears a little bit.

Could you tell us a little bit about your career and how you’ve ended up in this spot?

What’s your background and how did you end up here?

Joe:
Great question.

It’s really not obvious that I would have ended up in this kind of role when I started out 12 years ago.

I’ve had a very meandering career.

I think I’ve absolutely found myself in the niche that I love, and it’s because of all the opportunities that I’ve taken up to this point that I found it.

I started out in a very different type of marketing, which was more public relations, PR.

That was more trying to get our clients into newspapers and magazines when they were still a thing.

Then everything went online and it became trying to get mentions in online newspapers and journals.

That’s still very much an industry.

It’s slightly smaller than it used to be.

I enjoyed it for a bit.

I really enjoyed the research side of that, but most of it I wasn’t really a fan of.

So I moved out of digital PR and moved more into mainstream digital marketing.

What I would describe as mainstream is mostly paid digital.

Those kinds of channels where you see advertising online.

Google, social media, affiliate links, all of those different kinds of advertising.

I moved into a company where that was what we served.

It was a big agency that had a creative department.

It had a copywriting department.

It had a huge account management team, strategists, and planners.

That gave me a really solid understanding of how all of those different departments work together to create a solid marketing campaign.

Because you can’t really do a marketing campaign without all of those different parts.

You need to have the strategists and planners to understand what the audience needs and balance that with the client objectives.

Then you have the creative and copywriting team, which now is being squeezed a little bit in terms of what they can do with AI.

But that would be instrumental for crafting the right message for your audience to create a good hook and then the experience.

Then the account management team would handle delivery.

They would manage all of those different moving parts as well as activate the different media.

So it gave me a very good, rounded understanding of how those different parts move.

Whenever I speak to students who are wondering which kind of marketing they might like, I still think trying to go into an agency where there is a variety of disciplines at play is a good idea.

You can see how the different parts move together and how they orchestrate.

So don’t specialize too soon.

Try and be a generalist before you specialize.

I think that’s a good strategy.

Then when I moved away from that, I started to go into slightly more strategic roles, looking at more user experience and digital strategy.

My most recent role before where I am now was in an ad agency that had a very specific CRO team.

They had a whole team that was just looking at CRO and experimentation.

I stayed there for three and a half years.

Now I’m doing the same thing at a different agency.

It’s been a career of being a generalist and then specializing over time into my niche corner.

Along the way, I’ve made the most of those opportunities to understand what I get the most out of and what I enjoy the most.

Paying attention to those signals is important.

Zach:
It’s such an amazing experience that you’ve had.

You’ve had all these pieces of the puzzle along the way.

I think that’s such good advice to be a generalist and learn as much as you can about how these businesses operate and then specialize later on.

Given all of that experience that you’ve had, what’s the most surprising thing you’ve learned about human behavior?

You’ve been in these roles where you’ve done all this testing and met all these different people across all different facets of the business.

What’s the most surprising thing that you’ve seen in people and human behavior?

Joe:
That’s a really good one.

I would say the most enjoyable aspect of the job for me is probably getting to speak to a lot of people that I would never normally get to speak to.

For example, we have a client whose products and services are holiday adventures, tours around the world.

We undertook some research for them trying to understand the way that their customers think.

We spoke to 10 of their customers.

I had half an hour with each of them.

We’d speak with them about their experiences with the company, but also just about their life and why they chose this as a way to spend their money and use their time.

It means that we were speaking to people halfway across the world.

Some in Australia, some in South Africa, and other places.

You get to hear these amazing stories about their lives.

You only get half an hour with them, but during that time you get a completely different perspective.

I would have no business speaking to these people in my normal life.

It would be totally bizarre.

But I’m really glad I do get to because they always have such interesting stories.

I guess the most surprising thing is that people will always surprise you.

You can think that you know how other people might think.

The great lesson that I’ve learned is that the more experience I get, the more I realize I don’t know.

There’s almost an inverse to what you would expect.

The more specialized you get and the more knowledge you get, you’d expect to feel more confident that you have a good grip on how the world works.

Actually, it just reveals more and more of my own ignorance the more people I speak to.

I think that’s quite fun.

It’s a nice realization.

Zach:
You started to touch on a project that you guys are working on or have worked on recently.

It doesn’t have to be that one and obviously without giving away any trade secrets, but is there a project you’ve worked on recently where you’ve seen a major impact?

It could be sales or significant adjustments to a website or paid advertising.

Anything that’s had a major impact?

Joe:
There are a few I could mention.

I’ll mention a couple off the top of my head.

One is for the same company I mentioned, The Natural Adventure.

They do holiday adventure tours.

We work with them on optimizing their website, but also their advertising.

We were redesigning their whole checkout journey on their website.

One of my teammates, Josh, took it upon himself to review all sorts of different data points.

He was looking at user behavior data through Google Analytics as well as Microsoft Clarity.

He took screenshots and screen grabs from three different competitor websites and did a heuristic analysis of what kinds of rules and trends he was seeing in the competitive landscape.

He did user testing.

So speaking to actual customers and asking them how they get on with the website, watching them as they go through the funnel.

Then he combined all of that data and did what we call a page teardown.

That’s where you essentially annotate a page through Miro, which is kind of like an infinite whiteboard tool.

You literally take it to pieces and write down everything you think the website could be doing better.

Then we do a workshop around that with the client, get their buy-in on our changes, redesign it, and launch the new page as an A/B test.

That whole project took about six months because the website doesn’t get a huge amount of traffic, so it takes a while to get the data through.

But the outcome was that the number of people purchasing on the website went up by 20%, which is a huge number and way more than we would normally expect to see from an A/B test.

In terms of what they’re getting back from the rest of their marketing, think about it this way.

We’re spending loads of money just getting traffic to the website to begin with, but we’re getting way more from every visit now.

There were so many problems with the checkout that we hadn’t really realized before, and we were fixing those with the revised design.

That was probably one of the biggest wins we’ve had with a client in recent history because of the impact it’s had on their bottom line.

But then there are some smaller tests as well.

There’s almost a bit of a joke in the CRO industry about changing the button color to see if it makes a difference.

We did that with a client who had quite a boring website in terms of the color scheme and color palette.

It was also an industry that was really niche.

It was super high-tech and legal, so it was like law and IT at the same time.

We decided to invigorate the page with a slightly more jazzy button color.

It genuinely did have quite a big impact.

It had, in the region of, I think, a 15% to 20% uplift in people signing up or clicking on that button.

Then the last one goes to show the value of speaking to real people and customers.

Again, we were doing focus groups with another one of our clients.

This one is an international airline called Icelandair.

They’re the national carrier for Iceland.

We were doing focus groups and interviews with people who have an interest in going to Iceland.

We were showing them creative assets from Icelandair’s brand library.

We were showing them stimulus from the brand library and saying, “What do you think? What do you think of these materials? Do they make you want to go to Iceland? Do they put you off?”

These were essentially photographs of the landscape with some copy over them.

The really interesting thing we got back was that more than, I think, three or four people out of about six or seven said they thought the images were generated by AI, even though they weren’t.

They were almost saying they looked too good to be true.

“These don’t look real because the internet is now flooded with AI images.”

That insight ended up becoming part of a much larger campaign about AI essentially ripping off Iceland.

They had billboards outside Mark Zuckerberg’s house saying, “Please stop using AI for Iceland.”

They had one in Times Square in New York.

They had them all over London.

That’s a huge above-the-line marketing campaign with a lot of budget behind it, and that all came from one focus group with real people.

That’s the kind of stuff I love.

The small idea that can come from just speaking to a few customers can turn into this absolutely huge global campaign.

Those have been some highlights recently.

Zach:
That’s amazing.

That’s so clever.

Just using that piece of research, identifying that people aren’t sure whether this is AI or real, and then leveraging that as the creative in the next campaign.

That’s so cool.

You talked about the shopping experience or shopping cart experience being pretty bad for one of your clients.

Then changing button colors and things like that having a significant impact.

Are there common mistakes that you see pretty frequently with clients?

Is there one thing or multiple things you see a lot of your clients doing that are causing problems?

Joe:
Yeah.

There are still a few big no-nos that we see quite often.

Even big companies and some of the big high-street fashion brands are very guilty of some of this stuff.

The main one is probably just having a clear value proposition.

The first thing you see when you land on a website, it should be really clear who the company is and what value they’re bringing to you as a customer.

Why should I be interested in what you have to sell?

If you’re not immediately communicating that value within three or four seconds, people will just turn off.

Having a carousel on your homepage that’s constantly moving so that people can’t actually read what’s on it, they won’t pay any attention to it.

That’s a big problem that we see.

It’s actually got a name called carousel blindness because people don’t really pay attention to carousels.

They intuitively or unconsciously know that the information is going to move, so they don’t really pay attention to it.

We see that time and time again.

So making sure that your immediate impression is really impactful.

That also includes having a really clear and compelling call-to-action button that is above the fold.

What that means is above the bottom of the page, especially on mobile, so that people have a very good idea of what to do next when they land on your website.

They know that the next action is, “Explore our new range,” or “Get in touch to find out more,” whatever it is that you’re trying to compel them to do.

It needs to be as simple as clicking that button.

It needs to be really clear and direct.

Site speed is still an issue.

It’s hard to get around that.

That’s still what a lot of websites are penalized for.

But that’s a harder problem for a lot of people to solve because it’s usually a technical problem.

Then there’s a whole goldmine of social proof that a lot of people are sitting on, but they don’t really realize it is social proof.

When we talk about social proof, people are more likely to buy from you if they know that other people have done it before.

If you can give them a reason to buy into that herd mentality thinking, that’s useful.

That usually looks like Trustpilot reviews or some other kind of reviews.

But it doesn’t have to be.

It can be individual reviews.

It could be “As featured in” and then a bunch of well-known newspapers or outlets that you reference.

It could be reviews from the App Store.

It could be awards that you’ve won.

Or if you’re a legal or financial company, it could be some kind of certification.

There are lots of signals that people are looking for as soon as they land to give them reassurance.

Those are the common things that we’d usually advocate for.

Zach:
There are some common themes there with what you guys are doing and what you’re seeing across different websites.

I want to talk a little bit about the process.

If a client comes to you and says, “Help. Our website isn’t converting,” or they may not even know that language to use, but they’re saying, “We’re not getting the sort of sales we want to see out of our website or the conversions we want to see.”

Is there a process that you guys follow each time with a client?

Does that change?

If there’s a common theme, what does that look like?

Joe:
Yeah, there is for sure.

We try and guide them on a journey of becoming more mature when it comes to CRO and experimentation.

There’s almost the philosophy that it’s harder to start and get to one than it is to get from one to two.

Getting from standing still is the hardest bit.

That will include having a platform that allows these A/B tests.

You need to invest in a solution.

There are lots of them out there.

There are things like VWO, AB Tasty, Convert.com, and lots of other tools.

Picking one and investing in that is often the hardest part because it requires a leap of faith by the CMO or someone in the team.

You’re saying, “We’re going to put our money where our mouth is and invest in this.”

That’s a tricky bit.

Once you’ve got that buy-in, it’s about setting up a number of tests quite quickly to try and show that there is value in the program.

To show that it’s worth doing more.

That goes back to what I was saying earlier about what I call the great filter in CRO.

Most people have inflated expectations that the first test is going to win and not only win, it’s going to get them huge returns.

When it inevitably fails, they get really disheartened.

They completely lose faith in the whole program and want to chuck it in.

So you’re trying to get them past that and coach them past that.

Give them the right expectations.

It’s not going to be an immediate success.

You have to do a few to see the benefits and then scale from there.

We’ll often do what we call proof-of-concept tests to begin with.

Just get as many as we can live and make small tweaks here and there to get the ball rolling.

Then, as we get into the program, we’ll start to do research.

Research is the number one best way of improving your win rate.

Having hypotheses which are based in research will give you a much higher chance of actually getting a winning result.

We’ll do different types of research.

That includes qualitative research.

Speaking to people.

Also looking at survey feedback.

Any kind of voice-of-the-customer feedback.

More and more, we’re feeding that through AI so that we can shorten the process of analysis.

It still takes time, but we can do it quicker now.

Then there’s also quantitative analysis.

Looking at user behavior data as well as surveys, survey data, or large-scale feedback.

That could be something like Pollfish or exit surveys on a website.

We’re trying to triangulate what those problems are.

The more research data points we get that are pointing toward something being a problem, the more likely it is to actually be a problem.

If we’re doing one bit of research and it says something, that’s not a strong signal.

We need to do a few different types of research to triangulate that.

Then that research will inform the roadmap of tests.

That starts to gain momentum and build from there.

When you get the first big winner, you celebrate that as much as you can.

Get that test result far and wide within the business to raise the profile of what the program is doing and achieving for the company.

You’ll be surprised.

As soon as you get one big win like that, the number of people who start to pay attention and say, “Okay, there might be something to this.”

Then they might throw an idea into the ring as well.

A test they want to run.

From there it becomes easier because you’ve got a couple of advocates and champions around the business.

Then it starts to take on momentum itself.

Getting from zero to one is the hardest part because it takes a lot of grind to get there.

Zach:
Yeah, it’s a lot of convincing and trying to keep people on board until you see some return on investment.

Once you’ve got that evidence base, you can continue to grow from there and people have something they can grab onto with that ROI.

You talked about the importance of research.

I know we have a shared affinity for Daniel Kahneman’s research, and particularly his book Thinking, Fast and Slow.

I’m wondering whether you see those psychological principles he talks about in his book, anchoring, priming, framing, reciprocity, loss aversion.

Do you see these on a daily basis?

Are there examples where you see those at play, particularly within conversion optimization, but also more broadly across what you guys are doing at the top end of the funnel?

Joe:
Yeah, definitely.

The question I’m thinking about right now is whether a lot of those examples were led by those principles or whether it just so happens that you could retrospectively fit them within that rule.

But definitely.

Especially in the world of B2B, the biggest way of trying to get people to commit to speaking to you, because that’s usually the way to try and sell within B2B environments, getting them on the phone or getting them to see a demo, is by offering a free call, a free review, or a free consultation.

That’s where the law of reciprocity kicks in.

As soon as it’s free, or you’re giving value away, it feels almost like there’s an obligation to return that favor.

That’s definitely something in the world of B2B which is really powerful.

Loss aversion comes into play with a lot of e-commerce brands or travel brands.

The flash sale where you see timers on websites or see how many units are left on the website.

That’s a really strong loss-aversion play.

That can sometimes backfire if it’s not done with care because too many of those tactics can actually undermine the brand’s equity and the trust in the brand.

Then there’s framing.

In B2B or B2C, sometimes you’ll see subscriptions for coffee or something like that where they talk about cost per serving rather than the actual cost of the unit itself.

That’s a way of reframing the cost of that product as a much smaller number.

You might be paying £50 today, but actually it’s only £2 per serving.

That’s a really good use of framing.

Quite often you’ll also see in a subscription plan that the middle option is the one that’s most desirable.

They often give that slightly heavier weighting.

That’s a really good example of anchoring and framing as well.

Companies already know which one they want you to go for.

They’re essentially designing that page in such a way that you pay most of your attention to the middle option.

So always look at the middle tier when you’re looking at subscriptions and be aware of that.

Why do they want me to pick this one?

Is it the most profitable to them?

Zach:
That’s such a good example.

I do wonder whether personality type has an effect on what people choose in that scenario.

I’m such a cheapskate.

I’m always looking at the left side of the screen to try and find the cheapest option, even though they’re trying to frame it as, “This is the most value for money,” in the middle.

I wonder if personality has anything to do with those choices.

Those are great examples of framing.

The way they’re pricing per serving, that’s such an interesting example.

You talked a little bit about the tools that you guys use.

You mentioned Analytics and Microsoft Clarity.

Are there any other tools or software that you’re using on a regular basis that help in the area of CRO?

Joe:
Those two are the ones that come to mind because they have really universal appeal.

Most companies will have them, or at least a version of them.

They’re probably the most useful to become well versed in.

There are lots of qualifications you can do online for that.

But there are others that are slightly more specialized that we use in experimentation.

VWO is an example.

They have probably one of the largest customer bases worldwide, but they’re specifically A/B testing for websites and also apps.

In terms of other tools, the most common ones we’re using more these days are probably either ChatGPT or NotebookLM for processing vast amounts of customer data.

What we’ll do is clean that data before we put it into the platform to make sure there’s nothing sensitive or private in there.

Then we’ll essentially engineer prompts to try and understand trends and themes within that customer data.

The most common application of that is reviews.

We’ve been doing more of what I call review mining, mining for trends and themes within that customer data.

Man, it’s absolutely phenomenal how much you can do within that field now.

It’s stuff that we just wouldn’t have been able to do before because it would take too long.

An example is a client who had over 400,000 reviews.

That’s a phenomenal amount of reviews.

But that’s such rich information.

If we were to sit down and manually categorize each one of those reviews, not only by sentiment, so whether it’s positive, neutral, or negative, but also by the core themes within that.

For a clothing company, is that about quality?

Is it about fit?

Is it about value for money?

Is it about customer service?

There are all sorts of themes.

If we were to do that manually, it would literally take us months.

It would be the most tedious, boring task and nobody would want to do it.

AI has uncovered a whole new area of research for us.

A lot of brands are sitting on a goldmine of data that they’ve never used before in that way.

We’re really excited when we get a new client and say, “Have you ever done review mining?”

And they say, “No.”

We’re like, “Great, because we’re going to show you some really cool stuff with the review mining we can do.”

That’s all using either ChatGPT or NotebookLM.

That’s not necessarily because they’re the best tools to do it with.

There probably are other tools out there like Claude or other AI tools.

But at the moment, those are the best tools we’ve found to do that work with.

Those are the tools we’re frequently using more of.

Zach:
How are you guys doing the scraping?

Four hundred thousand reviews is a lot of reviews.

Are you manually grabbing those?

Are you using ChatGPT and other AI tools to go and grab those as well?

Joe:
That would come from the client.

That specific client had access to all of that data because they paid for Trustpilot and they also owned all of those properties within Google Business Profile.

They had lots of different stores around the UK.

They also had a paid subscription to Trustpilot, which means they can export all of that data as a CSV file and then share it with us.

It becomes trickier when you’re a brand that doesn’t pay for Trustpilot, because you can’t just manually download all of that information.

Then it becomes harder.

There are ways of doing it, but it requires scraping.

We only really work with data that has been downloaded from the client side and then they share that with us.

Zach:
That’s pretty handy if they’ve got access to that data already and you can just dump it in.

You sounded pretty excited about the use of AI, specifically for review mining, thematic analysis, and sentiment analysis.

Are there other trends happening right now within what you do that you’re excited about?

Joe:
Yeah.

AI gets lauded, and rightly so, for efficiencies and speeding up what used to be quite manual tasks.

But what I don’t think it gets enough plaudits or attention for is how it can help with coming up with really good ideas.

There was a study that I found recently.

I was at a conference watching a presentation, and they were talking about a recent study in Nature which was looking at how good AI is compared to humans at generating ideas.

Creativity, basically.

This was also a year ago, which in the world of AI is ancient history now.

It would have come a long way further than that.

But even then, the average person wasn’t as good at coming up with genuinely creative ideas as the AI was.

There were outliers.

You would find some humans who were way better than the AI, people who were really creative.

But in terms of an average, that completely floored me.

I’d always had this misconception that AI was only really good at doing repetitive human tasks, but wasn’t necessarily good at coming up with new things or innovative ideas.

I read that study and since then we’ve started using it more for idea generation based on first-party data.

Not only using things like voice-of-the-customer data, so reviews, but also all of the research that we do.

Speaking to real customers and then using the transcripts from those conversations and feeding that into the LLM as well.

Then using prompts that ask it to come up with ideas based on all of that data.

What I hadn’t really factored in is what it’s incredibly good at.

First of all, it’s unbiased in terms of the biases that we have from knowing the client, knowing what’s worked before, and knowing what’s worked with other clients.

We might have discounted an idea because it hasn’t worked for us at some point and therefore think it’s not worth pursuing.

It doesn’t have that bias in the same way.

It can also connect the dots between lots of different data sources.

It can surface trends within that data in ways that humans might miss.

It might pick up something that’s appeared in reviews, but also in transcripts, and perhaps in conversations within the chatbot on the website.

It identifies that theme and connects the dots.

What I’m really excited about is coming up with better ideas based on first-party primary resources that we’ve researched and gathered.

You could imagine a world where you’ve built what you might call a knowledge assistant.

You go to this knowledge assistant and say, “I’ve got this business problem or marketing challenge. Based on everything you know from our customer data…”

You could throw in your brand guidelines, your tone of voice, all of these things.

“How could you help with coming up with ideas to solve this problem?”

It could be a really good assistant in doing that.

That’s quite exciting, I think.

Zach:
It’s very exciting.

One question I want to ask you is, if AI is so good at these things, how do you see your role as an agency?

These things are becoming easier for businesses to do themselves.

I’ve got some thoughts on it, but I want to hear from you.

What do you think is your role as an agency if it becomes easier for businesses to do this themselves?

Joe:
I think effectively we are still shepherding our clients through a lot of this.

Even though this is definitely a trend, it’s not something I’ve noticed a lot of people are ahead of.

There still seems to be quite a long tail of people who are either AI skeptics or AI neutral.

They wouldn’t necessarily be the first people to jump on the bandwagon.

I see our role as an agency as being ahead of the curve and helping in the adoption and training of how to use AI and how it could be used for their own customer data and marketing data.

Being leadership in that strategy for how they can use it.

But generally speaking, as an agency, we recognize and are doing more of what we call “think for you” tasks rather than “do for you” tasks.

We can’t just take the efficiencies that AI provides us with and then sit on our hands and say, “We’ll pass the cost savings on to you.”

We have to use that time to do more strategic thinking and provide the client with better services that will help them achieve their goals with that added value.

The test then becomes whether the client thinks we’re doing that.

Do they still see us as a strategic partner rather than an executional, do-for-you partner?

That’s where I think we’re going to see a real divide over the next few years.

If agencies don’t make that jump from being an executional delivery agency to being a think-for-you strategic partner, there are going to be a lot of agencies that might not make it.

They become more expendable.

Zach:
That’s such a great way to frame it.

“Think for you” rather than “do for you.”

That’s so good.

I know we’re coming up on time here.

If you’ve got time, I want to ask you a few more rapid-fire questions.

Joe:
I’ll slurp down the answers then.

I’ll just be more brief, more concise.

Zach:
I just don’t want to take up all your day.

Is there a worst CRO mistake that you’ve ever seen?

What’s the worst thing you’ve seen on somebody’s website?

Joe:
There’s the famous Ling’s Cars, which, RIP, doesn’t exist anymore.

I don’t know if you remember that website.

Sadly, they’ve rebranded and updated the website.

It’s kind of lost its charm.

But that was the famous example, the kind of worst website ever.

It broke all of the rules, but was almost a fan favorite because it broke all of the rules and it knew it.

Other than that, it’s probably just really bad accessibility that you see time and time again.

I’ll probably call out a brand, and I’m fine with doing it.

Zara in the UK is the worst website I know of in terms of usability.

I’ve actually tried to call them out on it before and they’ve never said anything.

It’s massively style over substance.

It’s just bad.

Zach:
If we publish this publicly, maybe there’s no hope there for gaining them as a client.

Or maybe they’re right.

Is there anything that you’ve heard over your years of experience doing this that’s seen as best practice, but you think, “Actually, that’s not very good advice. You shouldn’t do that”?

Joe:
“Just make the button bigger” is a classic one, which definitely doesn’t really do anything.

The whole loss-aversion thing we touched on earlier also went through a bit of a craze.

A lot of e-commerce websites added that as functionality.

You might see a pop-up which says, “There are five other people with this in their basket right now,” or, “Ten purchased in the last 24 hours.”

Most people just know that’s BS.

They don’t buy it.

That’s a huge no-no these days.

If I see a client doing that, I’ll very quickly say that’s a bad idea.

Zach:
If you could A/B test anything in life, even outside of marketing, what would it be?

Joe:
Man, that’s a really deep question.

I’m tempted to say something really flippant and stupid like ice cream or where you would go on holiday.

Let’s go with holiday destinations.

Half of me could go to New Zealand and the other half could go to Canada, and I’d see which one I prefer.

Zach:
Very good.

I like it.

If you had five minutes to improve a website’s conversion rate, what would you do?

Joe:
I’d rewrite the headline.

That’s what I’d do.

I’d probably rewrite the copy on the main CTA as well.

That’s all I’d do.

Zach:
Apart from hiring Joe and his team at Launch to work on their website, what should students remember from this talk heading into their careers?

What should they remember about conversion optimization and experimentation?

Joe:
I would say that if you’re interested in people, the way people think, and also scientific methods, experimentation might appeal to you.

The research side might appeal to you.

But it probably won’t for everyone because it’s quite niche.

I’d say, as I said earlier, don’t specialize too early.

Remain open-minded.

Be a generalist.

Get as much experience as you can.

Then whatever seems to bring you more joy, more pleasure, and more fascination, pay more attention to that and see if that works out for you.

I guess you could treat your career as an A/B test.

Go down a few different routes and see which one works out.

Zach:
Is there anything else you want to cover or any other advice you want to leave students with?

Joe:
I think that kind of covers it.

One thing, and I’m a bit biased here because we’ve talked quite a lot before, but I would say what most students don’t realize is that lecturers like yourself have such a wealth of information in their heads.

Having a conversation with them, asking them their opinion, and getting their feedback is really valuable.

I didn’t do that enough when I was at university.

I was too scared.

I was kind of like, “I’ll do my work, I’ll hand it in, and I’ll see what my grade is.”

But you’re a lovely man and you’ve got a human side to you as well.

So make use of that time.

Have a conversation.

Zach:
That’s great advice.

I totally agree.

I’m often surprised at how few students take advantage of office hours and want to have a conversation.

That’s great advice.

Where can people learn more about you?

Joe:
Either go to LaunchOnline.co.uk, that’s our website, or you can find me on LinkedIn, Joe Johnston.

Either of those two.

Zach:
That’s great.

Joe, thank you so much.

I really appreciate it.

I always get a lot out of our conversations.

I really enjoy them.

I know students will get a lot out of this too.

Thank you so much, man.

I really appreciate it.