Different Methods of Testing, Optimizing, Predicting and Personalizing

Posted by | July 29, 2015 | Optimization | No Comments

I had a conversation with my dad today that sparked me to write this. He wrote me an email talking about how it seems like some of the products just “know who he is” and appear to know what he’s likely to do next. He’s not the only one who has asked me this in the past few weeks so I figured I would try to shed some light from my point of view.

There always seems to be a lot of talk about how important A/B testing is or new buzzwords like “optimization” or “personalization. Many of these trends often start from a key blog or industry expert calling out the ROI that it can provide to digital marketers. I’m all for testing and personalization, but it’s key to understand what exactly each is and what the benefits are. This is my attempt to bring it all back down to the layman.

Simple A/B Testing

There are many different types of A/B testing but what we call “simple A/B testing” is often the most common example. Simple A/B testing is the method of testing different variations against a control group. With the variations, they have equal weighting meaning that they are showed at random to users equally. Often times, tests are set up to have 95% of the users receiving the test (the variations) and 5% receiving the control (either nothing or what was there before). Simple A/B tests are often most effective when ran for a minimum of 7 days in order to gain sufficient data however the big key with run time is setting a determined end date for testing before hand.

Simple A/B testing is a basic method for testing and is often not necessarily the best option. The problem with this method is that the variations are equally weighted. This means that if a variation is not performing well, it will still get served up to users which may cause them to bounce. You effectively lose out on the potential to convert users. This is called the Opportunity Cost. Simple A/B should be reserved for basic things like button color, minor layout variants, etc. should you not have any other tool to use.

Bandit Testing Algorithm

This is where things get more interesting. An Bandit Testing Algorithm is a more sophisticated version of Simple A/B testing. With simple A/B testing, the weight of each of the variations are the same, meaning that if one variations isn’t performing well, you still serve that variation up. With Bandit Testing, as you explore the variations and their performance, you gain feedback as to how each variation does. What the bandit will do is start to weigh the winning variation more which means that it will serve up that variation more often (this is called exploiting). However, it will continue to test other less performing variations in order to explore the potential that they could still perform better (this is called explore). After the test has run for some time, it often becomes clear which variation is the winner. Most testing platforms implement this type of testing method.

The two most common versions of the Bandit algorithm are Epsilon Greedy and Bayesian Bandits. Many argue that Bayesian Bandits are the most sophisticated method for testing as they employ various statistic methods, such as probability distributions and probability densities, to find the best variation faster and more frequently.

The reason that Bandits are often considered better is that they dynamically update themselves based on prior testing knowledge. As the test runs and collects conversion data for the different variations, it starts to see which variation is performing better. It then enforces what we call “exploit & explore”. Exploiting is the system intelligently serving up the winning variation more frequently to find the winner faster. However, the system will continue to Explore losing variations every so often to ensure that it isn’t experiencing an anomaly and that the variation that is appearing to perform better actually is. You can think of it as a self-check.


With any product or service, retention of customers is always a key metric. It is much more expensive to acquire a new customer than it is to get an existing customer to return. It’s double as expensive when they churn! Since retention is a key metric for all companies, it’s important to really understand what it is. Retention is how often a user comes back to the product or service in a given time frame. In my opinion, retention is especially critical when paired with A/B testing as you get to measure the “stickiness” of your testing methods.

Monte Carlo Simulation

While not as widely used, Monte Carlo simulations help sophisticated testers predict the future with reasonable understanding. A Monte Carlo simulation draws out trends at random which is useful for running simple tests. However, for the industry, many will run daily Monte Carlos that update themselves based on historical data. For example, I may have a Monte Carlo that simulates out the potential trend for the next 30 days by aggregating the averages of the past 30 days. This gives me a decent look as to what the potential average trend might be and I may even be able to parse out an upper or lower limit from it as well. The testing industry uses Monte Carlo simulations to predict how a variation may perform in the future, how a particular segment group may change over time, and a few other unique prediction methods.

Regression Analysis

Understanding trends is a big value in the analytics and testing. Trends obviously help create insight into how an overall group may be transforming over time. A regression analysis can help users identify that trend. Regression analysis is a method for taking a group of data points or variables and estimating the relationships between them. The most basic method is called Linear Regression where you have a static line that basically creates an average based on the dataset. Regression analysis is useful for understanding the state of the dataset while being able to see how it may progress over time. It’s a very useful tool for analytics by providing insight based on history.

Segment Personalization

The ability to personalize based on a segment can be a very powerful feature when used properly. Segment personalization really means being able to deliver relevant content to a group of people who share similar attributes or interests. For example, let’s say that I’ve identified that on my travel website there are two types of visitors: Beach Going Travelers and Weekend Getaway Travelers. I have two segments that have very different interests, buying cycles, content consumption, and more. Through segment personalization, I’m able to deliver relevant content about different beach vacation packages to my Beach Going segment. These segments are often a mixture of both known and unknown users. Segment personalization is incredibly important since you’re delivering content that is actually relevant to a group which can reduce the likelihood of users in that segment to churn. The more aligned content is with a groups interest, the higher the potential for affinity to a brand, site, or app.

User Personalization

While segment personalization is at a higher level, user personalization is much more prescriptive. With user personalization, we’re personalizing content based on intimate known information about a user through different methods, such as dynamic messaging. For example, we might use dynamic messaging to send an email that says “Hey Ryan, you added this item to your cart. Check out these recommendations.” We’re using explicitly known data to interact with the user based on known attributes. This level of personalization is often referred to as a 1:1 conversation. We want to provide unique content and a different experience for a user  so that they don’t have to sift through weeds of information to get something relevant.


Taking personalization a step further is contextualization. Contextualization is delivering personal content within the context of the user. Much of the industry is turning to this however it is also one of the hardest to accomplish. For example, Starbucks knows that I’m a really heavy coffee drinker. They know that I usually start work at 9am based on how many times my device passes through a beacon (location identifier.). It’s 8:30am and I’m walking to work. As I get close to a Starbucks, I receive a push notification through my Starbucks app that says “Hey Ryan, it’s early and cold out. Come wake up and warm up with 10% off a mocha today!”. The message knows my name, my potential buying pattern, and the local weather. This is a very context driven message that starts to tug at different levels of consumer buying behavior. We can get this level of sophisticated personalization through 1st and 3rd party data sources that are collected from many different platforms. As we collect data from many platforms, we perform a method called “identity merge” which means that if you’re unknown in one platform, but known in another, when you identify yourself on the unknown platform we’re able to merge all of the known data about you into a unified user profile.

If you find this useful, feel free to comment!


Leave a Reply

Your email address will not be published.