Can Icons Outperform Text? Understanding the Role of Pictograms in OHMD Notifications

Although notifications increase the intake of digital information, enable proactive communication, reinforce a sense of connectedness, and help with task management [39, 45, 73, 75], they often reduce task performance, increase mental load, generate negative emotions, and disrupt social interactions [1, 7, 23, 40, 54, 57, 62, 89].

To balance access to notification information while minimizing attention costs [64], researchers proposed several strategies, which can be broadly classified into three categories: mediating strategies, which defer notifications until the user is more receptive to them; indicating strategies, which indicate the availability of the receiving party to the sending party; and, mitigating strategies, which change the device or presentation (modality) of the notifications to make them less distracting [2].

Of all the above strategies, mitigating strategies are the only type that provides notification information in a timely manner which can be particularly useful if immediate attention is required. Instead of delaying message appearance, mitigating strategies reduce distraction by re-encoding messages into an easier-to-understand representation to reduce the cognitive load. This paper investigates a particular mitigating strategy which is re-encoding partial textual information into icons.

Similar to other device notifications, OHMD notifications tend to interrupt the users’ primary tasks and reduce their work performance [59, 62]. Hence, to minimize interference with primary tasks when presenting information on OHMD, previous research has focused on different mitigating strategies, including the use of peripheral vision [16, 22, 47, 49, 60], adjusting presentation timing [70, 90], adjusting presentation modality [20, 24, 31, 59, 70, 90], and adjusting content placement [19, 53, 80]; with the latter including: content stabilization [33, 53, 56], alignment [80], and position [19, 53, 80]. In that regard, Klose et al. [53] and Fukushima et al. [33] explored content stabilization mechanisms by locking content through world, body, and head anchoring, and found that text readability improves with world and body anchoring; however, head anchoring is preferred for urgent texts such as notifications.

With regard to presentation modality, most of the works have focused on using multiple sensory modalities (e.g., visual vs. audio) [20, 24, 31, 70], while only a few have focused on using a single sensory modality [59, 90]. Consequently, this paper focuses on a single sensory modality, particularly the visual presentation modality, as it can be combined with other sensory modalities in a complementary manner [77] and is the predominant output mode in OHMDs [48]. Concerning a single sensory modality, previous work includes Lucero and Vetek's ‘NotifEye’ system [59], where they used animated butterflies as a cue for incoming notifications and allowed users to attend OHMD notifications with subtle interactions; however, their focus was on the pre-presentation stage of the notifications. In comparison, this study focuses on the presentation of the notification itself and seeks to reduce distraction and preserve communicative effectiveness by modifying the presentation modality using pictograms.

Pictograms, such as graphical illustrations, icons, and signs, are commonly used in daily navigation and digital applications in computing devices. They are also widely used in general communication, such as emojis/emoticons in messages [91, 105], communication in emergency scenarios [32, 103], and for people with intellectual disabilities [85].

Despite the prevalent usage of pictograms, there is no consensus on their effectiveness compared to the use of text. A large body of work, especially in the domain of traffic sign research (for reviews, see [8, 15]), favors the use of pictograms as they are not dependent on language [93], allow rapid recognition [27, 52], present information concisely [13, 66]. These advantages were demonstrated in a number of studies that found that pictograms were easier to use than verbal/text equivalents [14, 93].

On the contrary, another set of previous studies argues against the above notion [44, 87, 92, 101]. For instance, Warnock et al. [98, 99] investigated different notification modalities, including text and pictograms, in a desktop computing context for delivering home care reminders for older people who were engaged in a card-matching game; and the authors found that disruption during the game did not vary with the modality of the notifications. Moreover, Roca et al. [79] compared traffic signs in single words with pictograms and found that single-word messages were associated with better performance (farther reading distances) and required less visual demands (fewer glances and less glancing times) than pictograms. In addition, in the case of OHMDs, Tanveer et al. [90] delivered feedback on OHMDs during public speaking and found that the feedback in the form of text was more effective and easier to learn than pictorial feedback. As a result, using pictograms alone is considered less effective in communicating information as compared with the use of text [101].

Given this disparity in the literature about the effectiveness of using pictograms, we are interested in further investigating this problem to uncover the plausible reasons for such results.

While there are many kinds of notifications, in this investigation, we focused on calendar notifications as their high frequency in daily life (M ≈ 4, SD > 16 notifications per day according to Table 2 in [82]), uniform structure (sec 3.1.1), and high perceived value [51, 82, 95].

3.1.1 Notification structure. Calendar notifications could be thought of as being composed of two parts: 1) primary information (e.g., event/action) and 2) secondary information (e.g., time, person, or location). The notification “Meeting in 30 minutes” can thus be decomposed into “<primary info: meeting, secondary info: 30 minutes>”. To convert the text notification into icon-augmented notification, primary info was represented using icons, while secondary info was represented using numbers or text. An example is shown in Figure 1c . Note that for conciseness, prepositions, such as the word ‘in’ from the notification “Meeting in 30 minutes” were removed, and abbreviations were used in the icon-augmented notification, as our pilot studies have shown that they do not affect comprehension.

Three images labeled "a", "b", and "c" represents examples of regular text notification to icon-augmented notification mapping. Image "a" shows a typical calendar notification, and image "b" shows it's icon-augmented format. In image "c", the first column shows the regular text notification. The second column shows how the text notification is decomposed into primary and secondary information. The final column shows how primary information is represented by icons to create icon-augmented notifications. In the first row, the regular text notification "meeting at 4 pm" is converted to "meeting icon" and "4 pm".

3.1.3 Notification layout. As recommended by Debernardis et al. [26], all texts on OHMD were displayed in green color with a sans serif font (Roboto³). Our pilot study showed that text with a font of 50 sp⁴ under equal heights, an icon size of 50sp x 50sp, presented the best combination for space utilization and clarity on our OHMD device (Figure 2, sec 5.2). Notifications were displayed on the top-center position, as recommended by Chua et al. [19] for multitasking situations where primary tasks require central attention.

Two images labeled "a" and "b" represent the notification layout on OHMD. Image "a" shows how text notifications appear on OHMD. Image "b" shows show how icon-augmented notifications appear on OHMD.

In the first study (sec 5), the design of icon-augmented notifications was compared to regular text notification in a controlled multitasking scenario. The results showed that icon-augmented notifications outperform the regular text notifications in terms of task performance and reduced distraction. This finding provides evidence that the proposed icon-augmented notifications can be advantageous over text notifications; however, this doesn't address the second goal, as a careful examination of the study design revealed a potential confounding variable, filler words (e.g., linking words between primary info and secondary info, see Figure 1c). Due to the different affordances between icon-augmented and text-based notifications, filler words were retained in text notifications but not in icon-augmented notifications, which resulted in participants spending additional time reading regular text notifications. Thus, this confounding variable makes it difficult to draw any conclusion regarding the effectiveness of pictogram vs text.

To eliminate the potential bias, the filler words were removed from the text notifications, and a second study was conducted (sec 6, see Figure 5). Before conducting this study and to ensure that the resulting text notifications are comprehensible, a pilot study with 4 participants was conducted, and the results showed that the removal of filler words in the text notifications does not significantly affect their comprehension.

Examining the results of the study 2, it was observed that the previously observed statistical advantages in icon-augmented notifications from study 1 have disappeared, although the majority of participants still subjectively preferred icon-augmented notifications. The results of this study indicated that other than subjective feelings, replacing text with icons does not bring any statistically significant advantage.

While this finding is aligned with past studies that showed no advantages of pictograms over text, it can't explain why other previous studies have found advantages in pictograms. Hence, by carefully examining the design of the first two studies, two additional potential influencing factors were identified: content familiarity and encoding density.

Regarding the former factor, due to the difference in training and practice, participants were more familiar with text stimuli as they have been using text daily for many years; on the other hand, participants had less exposure to the icons used in the experiment and were not familiar with the meanings of all icons. Although we tried to narrow the gap with training and practice before the experiment, our post-experimental feedback revealed that participants were still not familiar with all icons used in the experiment, and such differences could potentially bias the results.

Regarding the latter, encoding density, which refers to the number of words an icon represents, is another potential influencing factor. For example, represents ‘Car’, which is one word, but represents ‘No Left Turn’, which is three words. Arguably, the higher the density, the more efficient the encoding process in the brain, which can affect the effectiveness of recognizing the notifications.

To understand how these two factors may affect the results obtained from study 2, a third study (sec 7, see Figure 7) was conducted by improving the participants’ familiarity with the selected icons and adding encoding density as another independent variable. In particular, instead of providing a predefined set of icons as stimuli, participants were allowed to choose their own icons, which is believed to help in increasing the familiarity with the stimuli.

The results of the study 3 showed that: 1) with increased familiarity, icon-augmented notifications have again demonstrated statistical advantage over text notifications, and 2) an interaction effect between presentation format × encoding density was found. A detailed analysis showed that the observed advantages in icon-augmented notifications are mostly attributed to the higher density conditions but not the lower ones.

Combining the findings of the three studies, a plausible explanation behind the reasons for the inconsistent results observed in the literature can be deduced; this explanation can be related to the fact that when comparing the performance of pictogram and text, many factors come into play. Pictograms can outperform text in multitasking performance, but with several conditions, which are: 1) the design of the pictogram needs to be intuitive and unambiguous, 2) users need to be highly familiar with the pictogram, and 3) the density of the pictogram needs to be high. If any of the conditions are not met, the advantages of pictograms may not be observed.

Examining the performance of icon-augmented notifications on OHMDs in real life can achieve a better understanding of how pictograms compare with text. Thus, a fourth study (sec 8) was conducted in realistic stationary and mobile settings to verify the generalizability of laboratory findings.The results largely confirmed the findings from the laboratory studies, where icon-augmented notifications, when carefully designed, can reduce distraction as compared to text. However, it also revealed additional factors that can influence users’ performance, such as the amount of external lighting.

The above is an overview of the current research, and the next few sections will present each of the above studies in detail.

All three controlled studies (Study 1, 2, and 3) were based on a dual-task scenario [72] to validate whether adapting pictorial representation reduces attention costs while retaining the content delivery. A vigilance task (i.e., an attention-demanding task, a perceptual monitoring task) was chosen as the primary task for which attentional control was measured [78, 84], and users were instructed to attend to the calendar notifications as the secondary task. Similar approaches have been used to measure the attentional cost of mobile phone notifications [89] and multitasking on OHMDs [53, 68].

Sixteen volunteers (8 females, 8 males, mean age = 22.7 years, SD = 2.5) participated in this study. All participants had normal or corrected to normal visual acuity with no reported color or visual deficiencies/impairments. All participants were from the university community who had self-reported a minimum level of professional working fluency in English and were avid smartphone users who received on average 51 (min = 30, max = 100) notifications per day; yet, none had prior experiences with OHMDs.

For all studies, the participants consented to the experiments and were compensated ≈ USD 7.25/h for their time. None of the participants in any study were involved in the subsequent studies.

The study was conducted in a quiet room under indoor lighting conditions to provide a consistent user experience and avoid confounding factors due to environmental interference [26, 34].

The primary task was displayed on a light grey background on a 23” LCD monitor (refresh rate = 60 Hz, resolution = 1920 x 1080 px) at eye level (see Figure 3a) and was designed using PsychoPy⁵ [74]. The stimuli of the primary task and notifications were presented at different depths to simulate attention switching between physical and virtual backgrounds. Hence, the participants’ eyes were set to be 70 cm away from the computer monitor, which differs from the focal length of around 1m [55] used by OHMDs. Moreover, the notifications were presented in an Epson Moverio BT-300 smart glasses [28] (Figure 3b), a binocular OHMD with 1280x720 px (30Hz) resolution display, 23° FoV, and a projected distance of 80 inches at 5m running on Android 5.1 OS (headset weight = 69 g). A BT-300 was selected as our OHMD since its functionality/features are a subset of the more advanced OHMDs (such as HoloLens2, Nreal Light, etc.); thus, its results could be better generalized to a wide range of OHMDs (details at sec 9.2.5).

Two images labeled "a" and "b" represent the study apparatus in the controlled experiments. Image "a" shows a participant wearing an OHMD clicking the mouse on target shapes appearing on the monitor. Image "b" shows a closer image of a participant wearing the Epson BT-300 OHMD.

In addition, a custom-developed Android application was installed on the OHMD to display custom notifications (Figure 2). A Python program handled the monitor's displayed stimuli, pushing notifications to the OHMD, logging user inputs, and synchronizing timings (see Appendix E for implementation details).

For the primary (vigilance) task, we adopted the shape detection task developed by Santangelo et al. [83] and reused by Mustonen et al. [68] to evaluate the visual task performance of an OHMD. This vigilance task was chosen for two reasons. First, the vigilance task is an established representative task that can simulate real-world usage of OHMDs in dynamic and unpredictable environments [68]. This is particularly relevant in situations where attention needs to be divided between the display (virtual content: notifications) and the environment (physical content: shape detection), such as walking in crowded areas. Moreover, this type of task can measure the degradation of sustained visual attention and perceptual monitoring capabilities when receiving virtual content, which affects gaze, attention, and situational awareness [68, 69]. Second, a controlled experimental task is needed for a fair comparison. Although real-world tasks have higher external validity, they often have many confounding variables that are hard to control.

During the shape detection task, the visual stimuli continuously morph between small (15 x 15 mm²) and large (30 x 30 mm²) white squares for 625 ms in segments that last 3750 ms. While the aforementioned small and large white squares are non-targets (Figure 4), a target shape that is either a vertical (15 x 30 mm²) or horizontal (30 x 30 mm²) rectangle randomly appears in 88.9% of segments, and no two rectangles appeared within 1875 ms of each other. Participants were instructed to press the left mouse button upon detecting a target rectangle shape at a time limit of 1875 ms, and the total duration of the task was four minutes to ensure sufficient time for assessing participants’ attention control [68].

The image represents how the white color stimuli appear on the gray color monitor. The non-target stimuli, a square, morphs between small and large sizes in cycles of 625 milliseconds. Then, stimuli randomly become the target stimuli, a vertical or horizontal rectangle. Participants have to respond to target stimuli by pressing their mouse button within 1875 milliseconds.

The secondary (notification) task was to attend to the calendar notifications on the OHMD. Six notifications were randomly displayed, during the primary task, with a minimum interval of 20 seconds between each notification and a display duration of 10 seconds, similar to the study conducted by Rzayev et al. [80]. Calendar notifications designed in sec 3.1 were used in this task.

A repeated-measures within-subject design was used to investigate the participants’ performance on primary and secondary tasks for three notification presentation formats: text notification, icon-augmented notification, and no-notification, with the latter being used as the comparative baseline. The experiment consisted of three testing blocks, each with a duration of four minutes, which were counterbalanced using a Latin square.

Each participant completed three blocks and received 12 notifications and 192 targets. They scored a minimum of 3 (out of 6) for recall accuracy and more than 72% hit rate at the end of each notification block. One participant's data was removed as an outlier since the hit rate deviated more than three times the standard deviation away from the mean. Table 2 indicates participants’ mean performance of measures.

Table 2: Performance in dual-task scenario (N = 15). Colored bars show the relative value of each measure for different notification formats. ^⋆ † ‡ represent significant post-hoc tests (p < 0.05). Here, Text = text notification, Icon = icon-augmented notification, No = no-notification, H = hit rate, F = false alarm rate, RT = reaction time, and RTLX = Raw NASA-TLX score.

The results show that icon-augmented notifications had a significantly higher primary task performance than text notifications. Hence, it can be deduced that the transformation facilitates multitasking since there was an increase in the primary task's performance while maintaining the secondary task performance (i.e., recall accuracy). Similarly, icon-augmented notifications significantly reduced Interruption while maintaining Reaction and Comprehension; additionally, they were preferred over text notifications. These results support the practical goal of this study (sec 4), the proposed icon-augmented notifications have advantages over text notifications during multitasking.

Nevertheless, it was observed that additional factors might affect the effectiveness of icon-augmented notifications based on the feedback from the participants. The presence of filler words in text notifications may have resulted in participants taking additional time to read the regular text formats as icon-augmented notifications used abbreviations while the regular-text version in text notifications did not. This feedback raised the question of whether the presence of filler words in text notifications affects its effectiveness during multitasking; thus, additional studies were carried out to examine the effect of filler words.

Twelve volunteers (7 females, 5 males, mean age = 23.4 years, SD = 2.7) participated in this study. They had similar backgrounds to participants from study 1 (sec 5.1), except for one participant who had previously used an OHMD for one hour.

Following the same notification design procedure in study 1 (sec 3.1.1), researchers selected icons to represent primary info and text/number for secondary info. To unify information content for the icon-augmented notification and text notification, the icon-augmented notification was transformed back to its text format without adding filler words (Figure 5). The three raters (who were in study 1) evaluated the icon-augmented notification and their transformed text notification to control for information difference and intuitiveness. If their content differed, the raters changed the text notification until full consensus was reached amongst the raters. Similarly, if any icon-augmented notification was not intuitive, the raters changed the icon.

Examples of regular text notification converted to icon-augmented notification and back to transformed text notification. The first column shows the regular text notification. The second column shows the icon-augmented notifications. The final column shows the transformed text notifications. In the first row, the regular text notification "meeting at 4 pm" is first converted to "meeting icon" and "4 pm", then converted to "meeting" and "4 pm".

All aspects were similar to study 1 (sec 5.4), except that the no-notification condition was removed during testing conditions. The format was fully counterbalanced using a Latin square.

Participants scored a minimum of 2 (out of 6) for recall accuracy and more than 68% hit rate at the end of each notification block. Appendix B.1 indicates participants’ mean performance of measures.

Surprisingly, there were no significant main effects or interaction effects for any quantitative measures (p > 0.05).

The results of this study revealed that the exhibit of the filler word is indeed a confounding variable. When filler words were removed, the performance between pictogram format and text format became overall comparable, indicating that text reduction influenced the increased multitasking performance and reduced distraction of icon-augmented notification in study 1. Yet, three caveats deserve additional attention. The first is that despite the lack of quantitative evidence support, the majority of the participants still preferred pictogram format over text format, indicating it is likely to still have some advantage (perhaps psychological). The second caveat is related to identifying a potential confounding variable, content familiarity, through analyzing the participants’ feedback. While all participants were familiar with the text content, not all participants were familiar with the (researcher-selected) icons. First, participants mentioned that less familiar icons were confusing and harder to recall, especially if they were not relevant to their personal lives. Second, some icons contained similar symbols, which confused the participants. Thus, the different familiarity among the participants towards the pictogram format and text format stimuli could affect their performance. The third caveat is related to the encoding capabilities of the icons, which affect the distraction from notifications, “when icons represent long sentences such as ‘valentine day’, it is way easier [to focus on the primary task] than text”. In other words, it was easier to recognize icon-augmented notifications when icons captured more words than the text notifications.

Twenty-four volunteers (11 females, 13 males, mean age = 23.7 years, SD = 3.8) participated in this study. They had similar backgrounds to the participants of study 1 (sec 5.1), except for three participants who had in the past used an OHMD for approximately three hours.

Since an icon can represent/encode either a single word (e.g., < birthday icon > represents ‘birthday’) or multiple words (e.g., < medical icon > to represent ‘doctor's appointment’); hence, in the context of this paper, encoding density (density) is defined as the number of words an icon represents. Accordingly, an icon representing one word (“birthday”) has an encoding density of 1, while an icon representing two words (“doctor's appointment”) has an encoding density of 2, etc. In this study, density has two levels: one and two.

Three human raters, two designers and one co-author, chose four representative icons with outline style (see Figure 6) for primary info from Google Material Icons, Flaticon website (premium license), and The Noun Project ¹¹, so that participants could select their preferred icon.

Two images labeled "a" and "b" represent the user's icon selection and corresponding icon-to-text mapping. Image "a" shows the user can select an icon out of four options. Image "b" shows the corresponding icon-to-text mapping according to the user's selected icon.

Using the user-selected icons (e.g., Figure 6a), 36 calendar notifications, each set comprising icon-augmented notifications and their corresponding (transformed) text notifications (see Figure 7) were designed similar to study 2 (sec 6.2). Half of them represented density one, and the other half represented density two.

Examples of regular text notifications, icon-augmented notifications, and transformed text notifications, along with the encoding density variable.

A repeated-measures design was used to investigate participants’ performances on primary and secondary tasks for the two notification formats and two densities. Thus, the experiment consisted of four testing blocks, text_one, text_two, icon_one, and icon_two, each with a duration of four minutes. The blocks were counterbalanced using a Latin square.

Each participant received 24 notifications and 256 targets in total during testing blocks. Participants scored a minimum of 2 (out of 6) for recall accuracy at the end of each notification block and yielded more than 67% for the hit rate. Figure 9, Figure 11, and Figure 10 indicate the mean performance for different measures (see Appendix C.1, Table 5 for details).

7.4.2 Icon selection. All participants found that icon selection helpful in recognizing and understanding the meaning of the icons. Only one participant suggested new icons as he could not find four matching icons from the given choices. Based on the survey results and the post-survey interview, all participants chose their icons primarily based on familiarity. Twenty participants also considered the clarity and simplicity of icons, while nine considered the pleasantness of icons. In comparison, two participants preferred detailed and skeuomorphic icons, two participants preferred thin outlines, and three preferred thick outlines. This indicates that people select icons based on icon properties as well as their personal aesthetic tastes.

Surprisingly, a massive disparity between the researcher-selected icons in study 2 and user-selected icons in this study was found as only 16% of the selected icons matched. For each icon, the percentage of participants who selected the researcher-selected icons in study 2 varied between 4% to 75%. Similarly, for each participant, the percentage of user-selected icons that matched the researcher-selected icons in study 2 varied between 8% and 53%. These results indicate individual differences in icon preferences and the lack of consensus on specific icons due to different mobile platform usage, and possible unfamiliarity issues that existed in study 2. For example, Figure 8 shows an example where < email icon > selection was agreed upon by most participants, while three different icons were selected as < meeting icon >.

Image shows two horizontal stacked bar charts. The top bar chart shows how image option one is preferred by the majority of participants to represent the "email" icon. The bottom bar chart shows that there is no consensus to represent the "meeting" icon, as participants’ preference is divided equally among three image options.

7.4.3 Primary (vigilance) task performance. Pictogram format had a significantly higher (p < 0.05) hit rate, while the false alarm rate and reaction time were comparable to the text format (p > 0.05). Moreover, the density two degraded the primary task sustainment of the text format.

• Hit rate (Figure 9a): Repeated-measures ANOVA after ART revealed a significant main effect of format (F_{1, 69} = 11.689, p = 0.001, $\eta ^{2}_{p}=0.404$, large effect [50]) and interaction effect (F_{1, 69} = 3.955, p = 0.049, $\eta ^{2}_{p}=0.187$, large effect), but no significant main effect of density. Moreover, there were significant (p < 0.001) simple main effects for text format and density two. Post-hoc analyses revealed that the pictogram format (M = 0.913, SD = 0.066) had significantly higher (p_bonf = 0.001, r = 0.698, large effect) mean hit rate than the text format (M = 0.881, SD = 0.081); and icon_one and icon_two had significantly higher (p_bonf < 0.05) hit rate than text_two. This also indicates that when an icon can represent multiple words, pictogram format can outperform text format.
  
• False alarm rate (Figure 9b): There were no significant effects.
  
• Reaction time (Figure 9c): There were no significant effects.
  

Three line charts represent the measures related to the primary task performance. They represent the comparison between text and icon formats for one and two densities in terms of hit rate, false alarm rate, and reaction time. See appendix Table 4 for more details.

7.4.4 Interruption. Icon-augmented notifications led to a lower cognitive load and perceived interruption than transformed text notifications.

• Unweighted NASA-TLX (Figure 10a): Repeated-measures ANOVAs showed a significant main effect of format (F_{1, 23} = 8.164, p = 0.009, $\eta ^{2}_{p}=0.262$, large effect), but no significant main effect of density or interaction effect. Post-hoc analysis showed that the pictogram format (M = 51.44, SD = 15.22) had significantly lower task load (p_bonf < 0.01, d = 0.374, small effect) than the text format (M = 56.93, SD = 13.99). The results for individual indices are given in Appendix C.2.
  
• Perceived interruption (Figure 10b): There was a significant main effect of format (F_{1, 23} = 19.068, p < 0.001, $\eta ^{2}_{p}=0.453$, large effect) and interaction effect (F_{1, 23} = 7.261, p = 0.013, $\eta ^{2}_{p}=0.240$, large effect), but no significant main effect of density. Post-hoc analyses showed that the pictogram format (M = 47.50, SD = 18.28) had significantly lower interruption (p_bonf < 0.001, d = 0.699, medium effect) than the text format (M = 60.48, SD = 18.54). Furthermore, icon_two had significantly lower perceived interruption (p_bonf < 0.05) than text_one and text_two.
  
• Distraction ranking: All participants, except two (92%) mentioned that the text format was most distracting since text required a longer time to read and understand and covered more space on the OHMD. Moreover, twelve participants who recognized the difference between the two text formats, mentioned that text_two caused the highest distraction. The remaining two participants felt that icons were more distracting as they needed to recall icon meanings to understand pictogram format compared to text format.
  

Two line charts represent the measures related to the perceived cost of interruption. They represent the comparison between text and icon formats for one and two densities in terms of unweighted NASA-TLX and perceived interruption. See appendix Table 4 for more details.

7.4.5 Comprehension. Overall, there was no significant difference between formats for immediate recall accuracy (p = 0.055), but the pictogram format had a significantly higher understandability (p = 0.020) than the text format. There were no main effects of density or interaction effect for any measures.

• Recall Accuracy (Figure 11a): There was no significant difference (F_{1, 69} = 3.796, p = 0.055) between pictogram format (M = 4.66, SD = 1.22) and text format (M = 4.29, SD = 1.15).
  
• Understandability (Figure 11b): There was only a significant main effect of format (F_{1, 69} = 5.667, p = 0.020, $\eta ^{2}_{p}=0.404$, large effect), and the post-hoc analysis indicated pictogram format (M = 5.48, SD = 1.15) had significantly higher (p_bonf < 0.05, r = 0.486, large effect) mean value than the text format (M = 4.79, SD = 1.75). This indicates that the pictogram format was easier to understand during the dual-task scenario than the text format when users engaged in an attention-demanding primary task.
  
• Understandability ranking: Fourteen participants (58%) mentioned that the pictogram format was easier to understand, easier to interpret, took less time to read, and was more concise. Out of the fourteen, six participants mentioned that the text format required more “mental effort” to understand than the pictogram format while switching their attention between primary and secondary tasks. Four participants mentioned that they felt no difference between the two formats, while the remaining six participants said that the text format was easier to understand since it was unambiguous and did not require any recalling of icons.
  

Three line charts represent the measures related to the comprehension and reaction parameters. They represent the comparison between text and icon formats for one and two densities in terms of immediate recall accuracy, understandability, and noticeablity. See appendix Table 4 for more details.

The results show that pictogram format sustained the primary task performance in terms of hit rate, particularly when pictogram format had density two compared to text format. Moreover, independent of density, pictogram format reduced the Interruption while maintaining higher Comprehension and Reaction.

Considering the results of this study and study 2, it can be concluded that user-selected icons improved familiarity, which in turn enhanced multitasking performance. Although, compared to Study 2, participants received additional time in this study to become familiar with both icons and texts through repeated self-exposure [86], user selection is the most likely factor contributing to these results. First, participants’ selection was based on familiarity, as detailed in sec 7.4.2. Second, as Shen et al. [86] have found, it takes considerable time to become familiar with icons, even with repeated exposure (e.g., more than 3 days with more than 30 minutes of exposure each day). Thus, the survey duration in this study may not have been long enough to achieve a high level of familiarity through repeated self-exposure alone. Furthermore, it can be concluded that the advantages of transforming text format to pictogram format do indeed depend on replacing texts with icons, as well as the icon familiarity. Hence, users should be given a choice to select icons for icon-augmented notifications to minimize interpretative ambiguities; however, we note that it would be challenging to cater to individual preferences. Similarly, when comparing this study with study 1, it can be concluded that making the text more concise also reduces the interruption of OHMD notifications.

Twelve volunteers (6 females, 6 males, mean age = 26.6 years, SD = 3.5) participated in the study. They had similar backgrounds to the participants of study 1 (sec 5.1).

Our system (a tablet computer) pushed one notification per minute onto the OHMD to avoid flooding participants with too many notifications and provide sufficient notifications to experience the differences between the two notification formats. This also simulated a situation in which users were engaged in a remote conversation [6]. Notifications were randomly displayed with a minimum gap of 40 seconds between each. The random presentation mimics situations where users forget their upcoming events or the events’ notifications are not created by the user, such as shared calendars. The notifications were chosen from text_one, text_two, icon_one, icon_two with equal probabilities such that they alternated between text format and pictogram format.

The briefing of the participant, icon selection, and training were conducted similar to study 3 (sec 7.3.1); however, participants engaged in the training session while walking in a university lab. As the natural light and weather conditions vary, the time of the experiment was randomly assigned to increase the generalizability of the results. Thus, participants completed the study during the daytime (9 am-6 pm), where they first performed the navigation task and then the browsing task.

Participants walked the predefined route at their comfortable pace while wearing the OHMD. The OHMD was equipped with removable lens shades during the navigation task to ensure the visibility of notifications in outdoor environments. The experimenter walked a few meters behind, carrying the system that automatically pushes notifications to the OHMD. Moreover, to ensure the participants’ safety, the experimenter closely monitored participants and stopped them from engaging in dangerous behaviors such as jaywalking. Once participants completed the navigation task (16-20 minutes), they ranked the formats based on distraction, noticeability, understandability, and preference.

Subsequently, participants engaged in the browsing task (8-12 minutes) and indicated their preferences for the stationary setting. In the end, the experimenter conducted a semi-structured interview to capture the reasons for each choice and understand how participants attended to notifications during the mobile and stationary settings. Each participant completed the study in one session lasting 50-65 minutes.

Some differences in the perception of notifications for mobile and stationary settings were found. The codes resulting from the thematic analysis were grouped based on the task differences and measures used in study 3.

The real-life settings yielded similar results to the lab-controlled ones (study 1, and 3) for Interruption (perceived cost of interruption) and Comprehension (for Recall Accuracy). However, the Reaction (e.g., noticeability) was affected by lighting conditions, and Comprehension (for perceived understandability) was affected by the complexity of the primary task. Specifically, outdoors, when external lighting became high, text format and its longer length became more noticeable than pictogram format.

Moreover, primary tasks during realistic settings were less attention-demanding; thus, more attentional resources were available to attend to the notifications than in lab settings which supports the finding that text format was perceived as more understandable than pictogram format during the browsing setting. A comparison of this result with that of study 3 suggests that when the primary task's complexity increased or needed higher attentional control, users familiar with icons could understand pictogram format better than text format.

Lastly, five participants felt that their OHMDs shook as they physically moved, which affected the legibility of the two notification formats. All of the participants felt that long texts’ legibility was more prone to shaking, while one participant felt that the pictogram format’s legibility was also prone to shaking.

The findings of our study suggest a plausible reason behind the observed disparity in literature (sec 2.3). Pictograms are likely to show advantages if they are highly familiar to the users and have an encoding density greater than 1; otherwise, they are likely to show no advantage or generate worse performance than text. This phenomenon is reflected in previous studies. Tanveer et al. [90] reported that participants found text feedback to be more effective and easier to learn than bar-like pictorial feedback on OHMDs during public speaking. In their study design, only one-word text was used; thus, the encoding density was 1. In addition, since the bar charts used in the experiments are not frequently used in everyday life, users may not be familiar. Thus, the text condition is likely to yield better performance. Similarly, Warnock et al. [98, 99] found no difference between the text and icon formats. In their work, the information contained in the text and icon-based notifications was limited to a single word, which equals an encoding density of 1. This finding is in line with the conclusion in study 3 that when the encoding density of icons is 1, both formats have a similar level of distraction for user-selected icons.

On the other hand, most studies favoring pictograms typically have a relatively small set of well-defined, commonly used pictograms with relatively high encoding density (much higher than 1, often close to 2 or beyond). For example, Ells and Dewar [27] compared traffic signs with their corresponding pictograms and found that the results significantly favor pictograms. Upon a closer examination of the stimuli used, we found that their study one compared eight pictograms with eight text messages. These pictograms were well defined and easy to understand with an average encoding density of 1.88, while their study two had fourteen pictograms with an average encoding density of 2.07. Similar analysis can be applied to Kline et al.’s [52] study as well, where they used four pictograms with an average encoding density of 1.75.

With a better understanding of the conditions in which icon-augmented notifications can have advantages, we can now discuss when and how to use them.

First, icon-augmented notifications are less suitable for delivering general-purpose notifications. This is because the vocabulary of general-purpose notifications is not restricted, making it difficult to find suitable icons for the diverse meaning a notification wants to express. Even if it is possible to find many icons, remembering them will be challenging for users, and it is unlikely that they will perform better than pure text-based notifications.

Nevertheless, icon-augmented notifications can be used in specialized domains; such as: notifying users of their personalized calendar or reminder events. Most of the calendar/reminder events are routine and recurring [51, 95]; thus, they can be easily represented using a few well-designed icons. Moreover, given the personalized context of calendars, as many of the events are set by the users themselves, it is also much easier for them to understand the meaning when they see a reminder notification.

In Figure 12, we suggest an approach to convert calendar events to icon-augmented notifications on OHMDs. Users first identify the frequent, recurring, or important tasks/events from their personal calendar, and for each event, the system (or user) identifies the primary information. The system then provides a set of icons for users to choose from or allows users to suggest/modify icons if necessary. Then, they can either directly go to step 4 to create icon-augmented notifications or use an intermediate scaffolding step (step 3) where both text and icons are used in a redundant fashion to help users to learn the associations first, then transit to the more concise version of icon-augmented notifications later.

The image describes the guidelines for converting text notifications to icon-augmented notifications. It uses calendar events as an example scenario. After identifying the primary information of the calendar event, it is represented by an icon. Then users see scaffolded icon-augmented notifications where icons are shown side-by-side with their text meanings until users get familiar with the icons. Once users become familiar with icons, they only see the icon-augmented notifications.

In addition to calendar events, OHMDs are used in medical, navigation, and manufacturing domains [67, 97], where users need higher levels of attentional control. In such specialized domains, it is easier to define a set of well-designed icons (e.g., [10]) that become highly familiar to users through repeated use. In such cases, we envision that icon-augmented notifications can ease the notification handling on OHMDs. Nevertheless, there are a number of additional considerations when applying icon-augmented notifications in practical applications.

9.2.4 One issue related to mobile OMHD usage is external brightness. To overcome the external brightness, designs can increase the salience of icon-augmented notifications in outdoor environments. For instance, borders can outline notifications (e.g., Figure 13), or colors can contrast/blend with the environment (e.g., [4, 25]).

Four images labeled "a", "b", "c", and "d" represent how the external lighting affects the noticeability of OHMD notifications. Image "a" show how a text notification appears on OHMD in indoors. Image "b" shows how a text notification appears on OHMD in outdoors. Image "c" shows how an icon-augmented notification appears on OHMD in indoors. Image "d" shows how an icon-augmented notification appears on OHMD in outdoors and how its saliency can be improved by adding a border.