Lesson: Sales Funnel Optimization with Hiten Shah

Step #6 A/B Testing: How A/B testing can help inform your sales funnel

The data helps you identify where you kind of need to explore. So, taking the example from before, if you realize that not enough people are purchasing but lots of people are visiting your website, and it's an e-commerce website where you're selling something, you would use surveying to figure out and learn why people are not doing what you want them to do, and why the people that did do it, did it. So, if someone purchased, I might just e-mail them right away after purchase, a simple question that says, "Why did you buy?" And then, learn all those answers, hopefully tally them, organize them so you get the top three reasons why people bought.

Then, if they're not buying on your product pages, I might ask the question of, "Are you going to purchase today? Yes or no? If not, please, tell us why not,” and that can help you understand why people are not purchasing.

Then I would take all that information, I would obviously organize that, figure out the top three reasons why people are not purchasing, I would take all that information and then the next step, otherwise, again, this information is useless as well, I would go design experiments. This is where A/B testing comes in. Basically, what A/B testing is, is the idea that you can take parts of your traffic or your visitors and show them a different experience.

This is different than just changing things. So, if you're not running an A/B test and showing different people different experiences, you're not actually understanding whether you made an impact or not. So, let's put it this way, if you do one thing for one week, and then you do something different for the next week, that's not A/B testing. That's just making changes. The reason for that is, scientifically, your traffic, the people that are coming in your site one week and another week might be completely different, depending on if you had press or if you started spending more money in advertising or something like that.

So, that's why you want to be able to split what the people see, but at the same time. So, that's what A/B testing really is, it's the idea that if someone comes to a site, the technology, it's very popular now, determines which version, A or B, they should see. But it's all based on the same traffic, that’s why you need to do it at the same time. That's why you don't do tests where you do one thing in one week and you do another in another week because the variation in your traffic can greatly impact the result of your experiment.

So, you would use, back to the surveying, you would use what you learned from the qualitative surveying, and figure out how to design a second version of your product page. Then half the people you would show the current version, and half the people you would show this new version. The whole key is not to guess how to make that second version, it's actually be data informed, and data informed means the idea that you're actually taking the information you know about why people bought, why they're not buying, and then creating a variation of that product page, a different product page, with that information.

So, if you notice that people say, if you ask them, "Why aren't you buying?", or, "Are you going to buy today?" and they say no, and they tell you that, "I'm not buying because I don't trust your site, I don't trust it", that would mean that trust is a big factor in your business, or with your visitors, that's something that they care about, that's what I would call they're sensitive to. So, you would design a page that you felt conveyed more trust. There's trust seals you can use, there's different colors you can use.

So, it's really about being informed in that test, instead of just guessing. A lot of times you might be right that your intuition might be right that, like, “Hey, our site's not trustful, and that's why people are not buying,” but if they don't say that, and you don't learn that, you don't really know, and you don't know that that's the issue, or these 10 other things that you can think of intuitively. So, the surveying really informs how you should be designing the test, and what elements you can be manipulating.

From there, you would essentially run the test, and then your funnel and the metrics you have, which you've already done, if you segment them by people that fell into bucket A or B, you'll understand what your conversion rate is, what your revenue is, depending on what they saw. That's what the funnel optimization, that's basically the whole process, soup to nuts.

I go try to really figure out who do I need to talk to that might be my buyer and go talk to them. Talk to them in such a way, not like, "Hey, I'm going to go sell skateboard wheels," but more like, "When you buy skateboard wheels," in this specific example, "where do you buy them from? Why do you buy them? When do you buy them?”

You might learn that only people who skateboard two hours a day need new skateboard wheels all the time. That actually already starts to inform you, "Here is my target demographic. Here's the type of person that's going to buy." Then you'd go to try to understand more about those people. Where do they hang out?

One of my favorite sayings about just building things and marketing and business is that you truly need to know more about your customer than they know about themselves. When you can get to that point, then I think it makes it much easier to figure out what to build and what to do. A lot of the risks in your business and the assumptions you're making are either validated or you move on to a different idea.

For example, you might realize that there aren't that many people that spend two hours a day skateboarding. So the market's not really that big and I don't even think I can make a living on it. So immediately, you didn't put a website up, you didn't do any of that, but in a very short period of time you learned that this is not a viable business.

That's why I started by saying that you need to really think about who's going to buy it. In a lot of businesses, the biggest risk comes out of the purchase, the buyer, the revenue, the money, and working backwards on really figuring out what assumptions are you making about that that might not be true, or which ones do you need to actually test and figure out if they are true.

There is testing and there is educated testing. I would say that a lot of people know how to do testing now, but they are just guessing. They are using their own intuition and a lot of times they are what they call a H.I.P.P.O., which is a highest paid person's opinion, to basically create a test. But that just relies on someone's opinion, not necessarily on educated testing, which is the idea that you would rather learn about your visitors or customers, get in their heads and learn about them and use that to educate what that test should be.

In my experience, when you are just guessing it takes much longer to make improvements than when you are making educated guesses. That would be how I would define educated testing.

There are ways you can understand whether the results you are looking at of an A/B test or a price test is statistically significant. So those tools are very important. If for whatever reason you don't have enough data, you honestly have to make a gut call or get enough data.

I would say that at this point I tend to make 8 out of 10 calls based on the data and the scientific method and stats and those 2 out of 10 are just ones that are hard to call because they are just so close between A and B. The way I think of it is which one would make the people in the company or the customer or even the company itself feel better. But that is usually when I can't get a statistically significant result.