1. Introduction
A/B Testing is a randomized experiment with two variant: A and B. It’s a way to compare two version of a single variable, typically by testing a subject’s response to A against B, and determining which of the two variant is more effective.
A: experiment side; B: control side.
A/B testing is a methodology for testing product changes. You split your users to two groups – the control group which sees the default feature, and an experimental group that sees the new features. Examples of A/B testing include Amazon personal recommendations, ranking change in LinkedIn (e.g. whether to show a news item or a possible connection to a user). The goal of A/B testing is to design an experiment that is going to be robust and gives you repeatable results so that you can make a good decision about whether to launch that product or feature.
2. What A/B testing isn’t good for
AB testing is useful when testing changes on a single variable. It is not useful when testing one variable against another.
A/B testing is not good for testing new experiences. It may result in change aversion (where users don’t like changes to the norm), or a novelty effect (where users see something new and test out everything).
The two things with new experiences is a) having a baseline and b) how much time needs to be allowed for the users to adapt to the new experience, so you can say what is going to be the actual experience and make a robust decision.
Finally, A/B testing cannot tell you if you are missing something.
In these cases, user logs can be used to develop hypothesis that can then be used in an A/B test. A/B testing gives broad quantitative data, while other techniques such as user research, focus groups, human evaluation give you deep qualitative data
3. When did people first start to use A/B testing?
A/B testing originated in agriculture. In medicine, their version of A/B testing is clinical trials. The difference between clinical trials and online A/B testing is that in clinical trials you have fewer patients with a lot of information on them. In online A/B tests, you have millions of users but limited information on the users.