In Parallel

Distributional assumptions are important for hypothesis testing.

Adobe Target is a popular website optimisation tool. The tool can serve different versions of a website or mobile app to different people. Analysts can then assess which version performs better on specified measures.

This article is about the statistical testing that Adobe Target uses. I also look at why distributional assumptions matter.

A/B Testing

Digital analysts are often concerned with the ‘conversion’ from specified pages. For example, what proportion of users proceed from a page to a product application form?

User experience experts and analysts seek to improve digital experiences. This is to increase conversion and reduce conduct risks like mistaken sales.

According to its documentation, Adobe Target has run two tests:

• Difference: the difference of two independent proportions. The initial hypothesis is that the difference is zero and follows a t-distribution.
• Lift (ratio): the relative difference of two independent proportions minus 1. This time, the null hypothesis is the lift is zero and has a Normal distribution.

To illustrate, the first version of a page has a conversion rate of 1.0%. The second has a conversion rate of 1.3%. That is a difference of 0.3 points, and a lift (relative increase) of 30%.