A/B Testing in Email Studio

Areas to test email content True or False

A/B Testing

Can you A/B test this?

Subject line: Track which subject performed better

A/B Testing

Can you A/B test this?

Emails: Select different emails and track which one performed better

A/B Testing

Can you A/B test this?

Content area: Use an email and track which content area is better

A/B Testing

Can you A/B test this?

From name: Choose existing one or create and test a new one

A/B Testing

Can you A/B test this?

Send dates/time

A/B Testing

Can you A/B test this?

Pre-header

‘A/B Testing' tab - 'Test type' section

Under 'Test Management', in this section, you can choose Subject lines, Emails, Content areas, 'From' name, Send dates/time and Preheaders.

‘A/B Testing' tab - 'Send to' section

Under 'Recipients' tab: you can choose List/Group, data extension or data filter. Select the percentage for each part of the A/B test.

‘A/B Testing' tab - 'Recipients'

Under 'Recipients' tab: you select publication list, suppression list and exclusion list if you need for test.

‘A/B Testing' tab - 'Determine winner by' section

Under 'Winner' tab: for this section choose 1 of 2: 1. Higher unique open rate. or 2. Higher unique click-through rate. Then choose 'Evaluation period’ by [hours] or [days].

‘A/B Testing' tab - 'Start test' section

Under 'Send Management' tab: choose 1 of 3: 1. Immediately 2. Scheduled 3. Scheduled later

‘A/B Testing' tab - 'Send email winner to remainder' section

Under 'Send Management' tab: choose 1 of 2: 1. Auto 2. Manually

‘A/B Testing' tab - ''From' options' section

Under 'Send Management' tab: choose 1 of 2: 1. Select a 'from' name and address 2. Select a send classification

‘A/B Testing' tab - 'Send options' section

Under 'Send Management' tab: tick the checkboxes you need: Send Multipart MIME, Track all links within the email, Suppress the send from reports or Enable send logging to a data extension

‘A/B Testing' tab - 'Email notifications' section

Under 'Send Management' tab: add email address if an error occurred or if a winner is selected.

Yes

Is ‘A/B Testing’ comparing two different versions of a message (Subject Line A vs. Subject Line B) to see which performs better? Yes or No

Yes

A ‘control’ is a baseline group of customers who do not receive any marketing communication. Yes or No

‘Control’

Imagine you own a clothing store and have 10,000 customers on your email list:

• The Test Group 1 (9,000 people): You send them a 20% discount coupon email.

• The Test Group 2 (1,000 people): You send them nothing.

After one week, you look at the results:

• 10% of the Test Group 1 made a purchase.

• 4% of the Test Group 2 made a purchase (without ever seeing the email).

The Result: Your email caused a 6% lift in sales. Without the Test Group 2, you might have incorrectly credited all 10% of those sales to your email.

What is The Test Group 2?

‘A/B Testing’ or ‘Control’

Yes - 5% to Condition A and 5% to Condition B. Usually 5% to 10%.

Is this best practice for how many subscribers are put into each A/B test condition for More than 50,000 subscribers? Yes or No

Send 5% to Condition A and 5% to Condition B. This tests a combined 10% of your audience, leaving a large 90% "remainder" group to automatically receive the winning version.

Yes - 10% to Condition A and 10% to Condition B. Usually 10% to 20%.

Is this best practice for how many subscribers are put into each A/B test condition for Less than 50,000 subscribers? Yes or No

Send 10% to Condition A and 10% to Condition B. These tests a combined 20% of your audience, leaving an 80% remainder group for the winning version.

Finding A/B Test Status in Email Studio

Email Studio -> dropdown menu, click ‘Email’. In the top navigation menu bar, click on the ‘A/B Testing’ tab.

Once inside this tab, you will see a main dashboard workspace table listing all your tests. The operational states (such as Scheduled, Running, Winner Selected, or Completed) will be displayed under the Status column for each individual test row.

Unscheduled

In Email Studio’s ‘A/B Testing’ tab, what is the status for the test is created and saved but has no execution time?

Scheduled

In Email Studio’s ‘A/B Testing’ tab, what is the status for the test has a specific date and time set to deploy?

Initializing

In Email Studio’s ‘A/B Testing’ tab, what is the status for the system is preparing the target data extensions or lists?

Running

In Email Studio’s ‘A/B Testing’ tab, what is the status for the test emails are actively sending to the test groups?

Overriding

In Email Studio’s ‘A/B Testing’ tab, what is the status for, a user is manually changing the parameters or stopping the automated logic?

Winner Selected

In Email Studio’s ‘A/B Testing’ tab, what is the status for, the system has determined the winning variant based on your criteria (e.g., highest open rate)?

Sending to Remainder

In Email Studio’s ‘A/B Testing’ tab, what is the status for the winning email variant is being deployed to the remaining audience ?

Completed

In Email Studio’s ‘A/B Testing’ tab, what is status for, the entire A/B test process finished successfully?

Cancelled

In Email Studio’s ‘A/B Testing’ tab, what is the status for, a user manually stopped the test before completion?

Errored

In Email Studio’s ‘A/B Testing’ tab, what is the status for, the test failed due to technical issues, system errors, or data problems?

No. Marketing Cloud calculates the winning version using percentages (rates), not raw totals with Unique Open Rate and Unique Click Rate. Because the winner is chosen based on performance ratios, the statistical validity remains perfectly intact even if one group delivered slightly fewer emails than the other.

When system randomly distributes your data extension or list into Test Group A (1,000 records) and Test Group B (1,000 records).

As the emails deploy, the system evaluates exclusions. If Group A contains 11 records that are on your suppression list, those 11 are dropped, and 989 emails go out. If Group B contains 10 records on the suppression list, 990 emails go out.

Resulting in Test Group A have 989. Test Group B have 990.

Will this tiny variance ruin your A/B test?

Yes or No

Deduplicate Upfront: Ensure your source Data Extension does not contain duplicate email addresses.

Filter Bounces Early: Exclude known held or bounced subscribers before starting the test.

Run a Pre-Filter Query: If you have heavy exclusion criteria, run a Query Activity first to create a "clean" audience data extension, then target that clean extension with the A/B test.

When system randomly distributes your data extension or list into Test Group A (1,000 records) and Test Group B (1,000 records).

As the emails deploy, the system evaluates exclusions. If Group A contains 11 records that are on your suppression list, those 11 are dropped, and 989 emails go out. If Group B contains 10 records on the suppression list, 990 emails go out.

Resulting in Test Group A have 989. Test Group B have 990.

How can you Prevent Major Group Imbalances in your A/B test?