Skip to main content

 

GovDelivery

Using Bulletin Filtering Criteria for A-B Testing

Communications Cloud's filtering criteria allow you to perform an analysis that improves the performance of your email communications. Essentially, your organization can send two different versions of a bulletin to small sections of your subscribers, with different subject lines, images, content, etc. and use reporting tools to see which of the two bulletins is favored by your subscribers. You can then send the favored/better-performing bulletin to the rest of your subscribers to maximize the results of your communications.

Testing Your Email Bulletins

10/10/80 Tests

To send a test bulletin to a small sample of your audience prior to sending out an important email, we suggest sending the first version to 10% of your subscribers, the second to another 10% of your subscribers that were not sent the initial email, and then sending out the “winning” version to the remaining 80% of your subscribers that were not sent either of the prior two emails. To accomplish this, please follow the directions below.

Step 1:  Send out the first version of your email. Use the Random recipient selection filtering criteria type to send the first version to 10% of your subscribers.

AB_Testing_Step1.png

 

Step 2: Send out the second version by applying two criteria. First, use the Random recipient selection filtering criteria type and select 10% of your subscribers. Then add a Bulletin send activity filtering criteria type to select subscribers that were not sent the first version email you just sent out.

AB_Testing_Step2.png
 

 

Step 3: Now, you'll want to see which version of your email performed better, to decide which copy to send to the rest of your subscribers. You'll need to wait a few hours to be sure the recipients in your test groups have had the opportunity to receive and read your email. Then, go into Reports, and click Bulletin Analytics Report. Then click the Details tab to see a list of your most recently sent bulletins. Here, you can view the open and click rates for both of your test emails. If you don't see open rate and click rate in your columns, click on the Columns link to add more data to this view.

Once you determine which email performed better, per your filtering criteria, click on that bulletin from the Details tab. Under the Sent To: line, you will see the option to Create Copy for Resend.

AB_Testing_Step3.png

Step 4: To send the higher-performing email to the remaining 80% of your subscribers, you'll want to use two more Bulletin send activity criteria. You'll send your better-performing email to subscribers who were not sent either of the first two versions of your email.

Testing your bulletins this way can lead to better engagement and overall results for your organization. While it takes a few more steps, the performance improvements are often well-worth it.

50/50 Tests

You can also perform 50/50 tests using filtering criteria. A 50/50 test is one where you send half your recipient list one version of your bulletin, and then send the remaining half a second version of the bulletin:

Step 1:  Send out the first version of your email. Use the Random recipient selection filtering criteria type to send the first version to 50% of your subscribers.

AB_Testing_Step1.png

 

Step 2: Send out the second version by applying the Bulletin send activity filtering criteria type to select subscribers that were not sent the first version email you just sent out. This will send the bulletin to all the subscribers -- 50% -- that didn't receive the first bulletin.


 

 

Step 3: Again, after a few hours to be sure the recipients in your test groups have had the opportunity to receive and read your email, go into Reports, and click Bulletin Analytics Report. Then click the Details tab to see a list of your most recently sent bulletins. Here, you can view the open and click rates for both of your test emails. If you don't see open rate and click rate in your columns, click on the Columns link to add more data to this view.

 


 

Does your organization frequently perform A/B Testing? 

Communications Cloud's A/B testing campaigns make executing this method quick and easy. With this streamlined approach, you can view real-time test results for on the spot optimizations or view changes in engagement over time to inform long-term strategy. You can learn more about these features by reading the A/B Testing Campaigns article.