Online Controlled Experiments Terminology
- chetanrajchaudhary
- Apr 2, 2024
- 1 min read
A/B/n test: Multiple Variant Tests. These are also called Field Experiments, Split Tests, Bucket Tests, Flights, Randomized Controlled Experiments
OEC (Overall Evaluation Criterion): quantitative measure of the experiments objective. OEC must be measured in short term yet drive long term strategic objectives. These are also called Response or Dependent Variable in statistics. Other synonyms: Outcome, Evaluation & Fitness functions. Selecting a single metric as a weighted combinations of multiple objectives is highly recommended
Parameter: A controlled experimental variable that is thought to influence the OEC and other metrics of interest . Parameters are assigned values called levels. Multi variable tests, also called Muti Variate Tests (MVTs) evaluate multiple parameters (variables) together such as font size, font color . Experimenters can discover global optimums when parameters interact
Variant: A user experience being tested, typically by assigning values to parameters. In a simple A/B test, A & B are two variants, typically called Control & Treatment
Randomization Unit: A pseudo randomization process is applied to units (users or pages) to map them to variants. Units must be mapped to variants in persistent and independent manner ( a user should see same variant in subsequent units and assignment of users to variant should not tell anything about the assignment of another to its variant ). Examples of randomization units: Users, Sessions, Pages, User-Day. Randomization is very critical. Randomization does not mean haphazard or unplanned but a deliberate choice based on probabilities