Evaluating Ad Creative and Web Context Alignment with Attention Measurement

We developed an attention measurement system that predicts attention time from user behavioural signals. The training data is collected from eye tracking panels in which subjects read media in self-directed sessions while eye tracking data is collected using a gaze fixation model trained on facial images [13].

As the eye tracking data is collected, we track a large number of other signals about the user experience including environmental signals like the page structure, and behavioural data like scroll events and the position of the advertising in the viewport.

The signals collected along with the eye tracking data allow us to build and deploy an attention measurment model that can predict the attention time paid to specific ad units within a page. The attention model can be applied to all digital inventory that accepts the javascript tag.

Our model evaluation process focuses on the accuracy of measuring mean attention time for a specific ad format, over varying samples of inventory. To generate the performance statistics we run bootstrap sampling of the test data to look at the expected error when predicting the mean attention time over impressions for a specific ad format. Each metric is calculated by taking 500 impression samples 10000 times, and using the resulting distribution of mean error. The mean attention time measurement performance is summarised in Table 1. We see that the aggregate mean absolute error (MAE) for both formats is very low at 35ms and 48ms for banners and MRECs respectively. Furthermore, the $95\%$ confidence intervals for these values is constrained to be one the order of 100ms either side of the true value. We apply this attention measurement model across large scale digitial inventory to collect a data set of attention time paid to a range of advertisments over time. We use these attention measured impressions to investigate the impact of contextual categories of media on the attention paid to advertising.

The data used in this study was collected from a wide range of advertisers running broadly targeted campaigns between February and March 2022. It includes brands in the automotive, retail, travel and health services verticals. The data is broken into two one week time periods, the raw data was filtered such that we had impressions for every creative in both periods of time. This was a requirement of our approach so that we can study the continuity of the relationships between creative, context and attention across time.

In the analysis that follows we apply different levels of aggregation to the log level impression data, so that we calculate the mean attention time for a given creative in the specific context. Within each experiment we will filter the aggregated data to ensure that every sample point is derived from a least 50 impressions. This is done to ensure robust estimates of mean attention within a given category. Summary statistics for the dataset are shown in Table 2. Note, that due to the restriction outlined above, not every sample point will be in all experiemnts.

In order to derive contextual categories for each record in our dataset, we applied the Verity API (GumGum) to the URLs. We converted the results of the API response into a variety of features that describe the contextual categories derived from the iAB V2.0 hierarchy [4].

We identified the iAB category for which Verity had returned the highest score and converted that category into a set of labels for classification across the four Tiers of iAB V2.0. In instances where the classified category is from a higher position in the hierarchy, then the lower level tiers are given the deepest available category. For example, if we have a URL that is classified as iAB V2 category "Business and Finance" (a Tier 1 category) then Tiers 2, 3 and 4 will all contain the same value.

Alternatively, if we have another URL that is classified as "Marketing and Advertising" (a Tier 3 category that sites below "Business and Finance"), then the hierarchical labels would be assigned as follows:

• Tier 1: ["Business and Finance"]
  
• Tier 2: ["Business"]
  
• Tier 3: ["Marketing and Advertising"]
  
• Tier 4: [ Marketing and Advertising ]
  

In the above instance the deepest initial categorisation is "Marketing and Advertising" coming from Tier 3. This category is then inherited to any remaining tiers (as indicated by the italics at Tier 4 in this example). The idea of this scheme is that every page will always be assigned a categorisation across all four levels in the hierarchy, but sometimes the category at a given Tier will not be any more specific than the parent category. This approach ensures that when we make comparisons of the effectiveness of targeting at a certain Tier we are always able to include all data points. This approach is both more robust, in terms of the comparability of results, and pragmatic in the sense that when using these categories for campaign targeting a decision would always need to be made about whether a given URL satisfies the targeting requirements at a specific Tier. Additionally, this approach ensures that when we look at a deeper level in the hierarchy we will have at least as many distinct categories as the parent Tier.

The focus of our analysis is to determine whether the performance of each creative, in a set of contexts, remains consistent across the two time periods. Within each time period we aggregate the measured attention time over the impression level data to obtain multiple metrics for the reliability of attention time within a given context.

The attention metrics are calculated for a set of impressions I_p in a time period p. We first define a baseline for a creative c in period p as: ${}^c\mathcal {A}_p$. This is defined in equation 1.

Where the function A(i) returns the attention measured on impression i and 1^c(i) is an indicator function that tests whether impression i belongs to creative c, as shown in equation 2.

Similarly, we denote the baseline attention achieved by ads on pages classified as belonging to category t in period p as: ${}_t\mathcal {A}_p$. Defined in equation 3. Note, we use the notation t for categories as they are equivalent to topics, and this distinguishes them from the creatives.

Where 1_t(i) is an indicator function that tests whether impression i was shown on a page classified as category t, as show in in equation 4.

We denote the mean attention performance for a creative c, on media of category t within period p as ${}^c_t\mathcal {A}_p$. This is defined as shown in Equation 5.

Where we simply apply a indicator function $\mathbf {1}^c_t(i)$ that is a composite of Equations 2 and 4 to filter for inventory that belongs both to creative c and category t. These equations should illustrate that we make our evaluations by looking at the expected attention time across difference slices of the inventory.

Our analysis proceeds by ranking all topics that each creative c was exposed to according to the value of ${}^c_t\mathcal {A}_1$. We then look at the same creatives in period 2 and evaluate ${}^c_t\mathcal {A}_2$ for the top three ranking categories from the first period, as well as the bottom ranking category. This approach necessitates that each creative be exposed to a minimum of these same 4 categories in both periods. In the results that will follow we report the number of records in each trial, reflecting the number of aggregated records that fulfills these requirements.

Once we have the datasets in each period that meet these requirements we use multiple metrics to evaluate the impact of the contextual categories. Our primary metric is the percentage of instances in which the category specific performance ${}^c_t\mathcal {A}_2$ is above the creative's baseline expected performance ${}^c\mathcal {A}_1$. This is shown as % Over (the baseline) in the tables of results.

In addition we look at the mean difference between ${}^c_t\mathcal {A}_2$ and ${}^c\mathcal {A}_1$ over all creatives in the experiment. This is denoted \boldmath $\bar{\Delta }$ (or mean delta) in the tables and defined in Equation 6

The mean delta illustrates an expectation of the difference between attention time performance and the creative baseline, using a specific contextual targeting strategy. In addition, we show the difference expressed as a mean percentage (against the baseline) in the column $\mathbf {\bar{\Delta }\%}$. This can be interpreted as the expected percentage increase/decrease relative to the creative's baseline attention, when using that contextual targeting strategy.

In Table 3 we see the results of looking at performance on novel URLs for a creative, where the iAB V2.0 contextual categories of those URLs were present in either the top three or last ranking position in the initial period.

We see that, in general, targeting pages using categories deeper into the hierarchy improves the expected increase in attention through contextual targeting. However, the effect appears to plateau at Tier3, likely due to the small proportion of inventory that is uniquely classified at a Tier 4 level.

We show the change in attention lift across the dimensions of iAB Tiers and category ranking in Figure 1. This figure demonstrates the increased performance with the specificity of the iAB Tiers and the reduction in performance with the movement down category ranks.

We repeat the analysis restricting ourselves to impressions in the second period that involve URLS for which the domain was not present in the URLs for the first period. The results are shown in Table 4. We note that the number of records has dropped by more than half in this experiment, due to the difficulty of finding domains that were not present in the first period.

We show the change in attention lift across the dimensions of iAB Tiers and category ranking in Figure 2. This figure demonstrates that in the no domain repeat experiment we get a very similar overall pattern to the previous experiment. The primary difference being a higher expected lift at the rank one position, and a faster drop off in expected lift as we move down the ranks.

The restriction to unique domains in the second period has lifted the expected improvement of targeting the top performing category (Tier 3) from 135only occurs with the category in ranking position one. For the other ranking positions the results are mixed, even though the percentage amount is sometimes higher, the absolute amount can be lower, due to the fact that the baseline is calculated on the impressions that qualify for inclusion.

These result seems counterintuitive at first glance. Our current hypothesis about why this is the case is that it relates to the specificity of the content on a given domain. The no domain repeats study is biased towards data on less common domains. These domains may have a tendency to be more subject specific, i.e less tendency for discussion of multiple topics.

Additionally, we note that the percentage of instances where the category targeting delivers an improvement over the baseline drops when we look at unique domains. This drop is approximately 3the 3rd ranked category it has dropped from 80targeting new domains with the 3rd ranked category offers a marginally better than random chance of delivering improved attention.

We estimated p-values for the significance of the proportion of URLs within a given category that outperform the baseline. We did this using a binomial distribution, under the null model assumption that the relationship between the attention on a URL and its contextual category is random. Hence, the p-value can be estimated as the probability of obtaining the observed number of successful improvements over the baseline as samples from a binomial with a probability of success equal to 0.5.

The computed p-values were extremely small, starting at 10^{− 62} for first rank of Tier 1 in the No URL Repeats experiment, 10^{− 24} for the No Domain Repeats experiment. The p-values increase gradually as we move down the ranking of the categories or at deeper levels in the iAB hierarchy, but they always remain below a significance level of 0.05.

A more stringent analysis would apply a Bonferroni correction to account for the multiple hypothesis tests in these tables of data. Even with this correction, the vast majority of results are many orders of magnitude below the significance threshold. However, under this criteria we would reject the third ranking category within the No Domain Repeats experiments. However, as the observed data is overwhelmingly statistically significant, and these values are consistent with the hypothesis being true, but suffering from a diminishing effect size, and reduced sample sizes, we included the values for the third ranking categories in the table.