How to choose population when Designing experiment


Assuming we are doing an A/B testing. 

You have to decide who you are targeting in your users, i.e., the target population. Such as the U.S., Europe, or globally.

Reasons why you might determine the target population in advance in order to restrict who sees your experiment:

Reason 1: If you are running a feature and you are not sure if you are going to release it and it is a pretty high profile launch, you might want to restrict how many of your users have actually seen it. So that you don’t get any press or blog coverage.

Reason 2: You might not be sure that your feature works across language, browser type, or platform, and you may want to just restrict it to certain users.

Reason 3: You may not want to dilute the effect of your experiment across a global population. So if you are analyzing an experiment for the first time, and it only affects english-speaking areas, you may want to actually do your analysis specific on English, and be able to ignore the rest of the population.

For an example of how experiment result is diluted when change the population, click here.

error: Content is protected !!