Applying Machine Learning to Gaze Data in Software Development: a Mapping Study

The demographics of the selected papers are listed in Table 1. Almost all papers (9 out of 10) were published in 2018 or later, with a peak in 2021 (3 papers). In terms of venue distribution, 8 papers were published in a conference or a symposium, and 2 in journals. The International Conference on Program Comprehension (ICPC) is the top venue with 2 papers. The majority of the papers (7) presented a study that included data collection, while 3 papers reused an existing data set. The majority of papers (8) report on using educational material as stimuli, while the remaining papers (2) report on using code snippets from either open-source software (OSS) or closed-source software (CSS). When it comes to the choice of stimuli, the same type of material, e.g., educational material, is commonly used for both novices and experts, but there might be an increase in task difficulty or code complexity (or experts may get additional tasks). For studies with only practitioners, researchers tended to employ CSS material for the purpose of capturing realistic gaze behaviors in the workplace.

The majority of studies (6) were carried out in a laboratory setting, with a few (2) also including a workplace setting. One study was carried out in a classroom, and one study was carried out in a workplace. The studies in a mixed setting are usually those contrasting novices with experts in programming expertise, where novices are usually university students and experts are practitioners in the industry. Depending on the goals of the study, sometimes researchers asked both groups of participants to perform the tasks in the lab, while in some other cases, researchers conducted the experiments with experts in their offices.

The majority of papers (8) report using students as participants, which in a number of cases (6) were mixed with researchers and practitioners. The remaining 2 papers report using only practitioners. The number of participants ranges from 15 to 157, with a mean of 48, a median of 30, and a standard deviation of 45. Finally, half of the studies (5) used additional sensors; for Electrodermal Activity (EDA), Galvanic Skin Response (GSA, coded as a type of EDA), Electroencephalogram (EEG), Heart Rate Variability (HRV), mouse, and camera. The most common additional sensors were EDA (3) and EEG (2).

To address this research question, we collected data on the software development activities studied. The top row of Table 2 gives an overview of the activities we found and the bottom row in the same table shows the total number of occurrences of an ML technique being applied to a data set gathered from this activity. Code comprehension is the most studied activity (12), followed by code review (2), pair programming (2), and visualization of debugging gaze data (1).

Beyond activity, we also collected data on the purpose of using an ML technique. Across almost all papers (9 of 10), we found the purpose to be some kind of prediction task, summarized by the top row of Table 3 and counted in the bottom row of the same table. One paper, not shown in the table, used graph embeddings for mapping graphs into a 2D space. Note that multiple papers describe studies where several ML techniques were applied to the same data set [Abbad-Andaloussi et al. 2022; Ahsan and Obaidellah 2020; Fritz et al. 2014; Villamor and Rodrigo 2018]. One paper may also describe more than one prediction task [Lee et al. 2018]. As such the totals in these tables (17) exceed the number of papers (10). The most common ML task was to predict programmer expertise (6), followed by task difficulty (4). After that we saw prediction of mentally demanding code (2), success of pair programming (2), quality of code review (1), affect in code review (1), and code reading ability (1).

To address this question, we consider what ML techniques are applied on gaze data and how. From the heatmap presentations in Table 2 and Table 3, we see that Support Vector Machines (SVM) and Decision Trees are the two most commonly used techniques, followed by Random Forest (2), K-nearest neighbor (KNN) (2), and Naive Bayes (2). The preference for the top two techniques and Random Forests is due to their capability to handle large data sets [Lee et al. 2018; Vrzakova et al. 2020], overfitting [Al Madi et al. 2021; Lee et al. 2018], and effective ranking/selection of features [Al Madi et al. 2021; Vrzakova et al. 2020], while the preferences for KNN and Naive Bayes is due to simplicity [Fritz et al. 2014] and/or explainability [Hijazi et al. 2021].

To dig deeper into how the ML techniques are applied, we consider the ML techniques from a perspective of a model taking input and providing output, summarized in Table 4. The listed models correspond to the ML techniques listed earlier in Table 2 and Table 3. In the input column, we convey our interpretation of the details from the descriptions provided in the papers along with examples for clarification. We find a fairly high degree of heterogeneity, which may not be surprising given the available granularity/properties/dimensions of gaze data. However, in general, we see a tendency to use raw gaze data in combination with embedded feature selection algorithms (e.g.,  [Abbad-Andaloussi et al. 2022; Lee et al. 2018]) to help pinpoint the prominent features for a specific task. In cases where several sensors are used (e.g., EDA, EEG) or data is gathered via other methods (e.g., answers to a questionnaire for participant performance assessment [Ahsan and Obaidellah 2020]), the data from the other sensors are incorporated in the training data set. Further, some researchers also used analysis results or derived metrics (e.g., CRQA attributes [Villamor and Rodrigo 2018]) of gaze data as the input for training their models.

We address this research question from two aspects: limitations and challenges. We constructed the following themes from the limitations reported in the set of selected papers (not all papers reported limitations), sorted on occurrence in papers:

• Limited sample size (4 papers): the size of the sample is not large enough (perhaps despite being representative [Al Madi et al. 2021]) and thus may impact the applicability or generalizability of the model [Abbad-Andaloussi et al. 2022; Fritz et al. 2014; Vrzakova et al. 2020].
  
• Unbalanced data set (3 papers): skewness in sampling, insufficient or absence of representation of certain cohorts in the data set [Fritz et al. 2014; Vrzakova et al. 2020], e.g., lack of samples of professional programmers [Al Madi et al. 2021].
  
• Binary categorization (2 papers): specifically refers to the potential risk of over-simplification of programmer expertise and the task difficulty with a binary categorization method, e.g., novices vs. experts [Hijazi et al. 2021], and easy vs. hard [Fritz et al. 2014].
  
• Information trade-off (1 paper): loss or compromise of certain information due to the use of an ML technique, e.g., the nuanced differences between two gaze trajectories became difficult to identify after having been mapped into a 2D space in the work by Zhang et al. [Zhang et al. 2019].
  

Similarly, we created themes based on the challenges reported in the paper, as follows:

• Experiment management: the inherently difficult-to-control conditions of an eye-tracking experiment, potentially interwoven with other advanced sensors (e.g., EDA [Abbad-Andaloussi et al. 2022; Fritz et al. 2014], EEG [Fritz et al. 2014], or HRV [Vrzakova et al. 2020]), and sensitivity to environmental variables such as illumination, the Hawthorne effect [McCambridge et al. 2014] (participants may attempt to act in a manner pleasing to the researcher), and learning effects while performing the tasks [Fritz et al. 2014]. In addition, a mismatch between a laboratory setting and a natural work environment is most likely unavoidable.
  
• Generalizability and validity of ML models: ML models by nature are vulnerable to bias embedded in the training data set [Ahsan and Obaidellah 2020; Fritz et al. 2014; Vrzakova et al. 2020] and overfitting when trained on a small data set [Abbad-Andaloussi et al. 2022; Al Madi et al. 2021; Fritz et al. 2014; Vrzakova et al. 2020]. This sometimes calls the validity and generalizability/applicability of the trained models in question. However, some researchers deliberately chose certain ML algorithms, e.g., Random Forest [Al Madi et al. 2021; Vrzakova et al. 2020], over others to mitigate this problem. Regardless, researchers still stated that due to the lack of diversity in their data and a small-sized sample, further validation [Ahsan and Obaidellah 2020; Lee et al. 2018] or in-situ evaluation [Hijazi et al. 2021; Vrzakova et al. 2020] with a larger or unseen data set is needed or recommended.
  

Additionally, the challenge of getting access to professional programmers was also mentioned [Abbad-Andaloussi et al. 2022], as well as the challenge of finding the optimal parameters to derive metrics for training [Villamor and Rodrigo 2018].

Below we synthesize the future work stated in the studied papers as opportunities, supplemented with our own insights (sorted after occurrences in papers):

• Extension of application (4 papers): the possibility of applying the trained models to a similar sub-area [Lee et al. 2018; Vrzakova et al. 2020; Zhang et al. 2019] or a neighboring field rather than software development, e.g., computer science education [Hijazi et al. 2021].
  
• Prediction task expansion (4 papers): the aspiration to expand the prediction of the trained model to cover other similar or finer-grained tasks [Ahsan and Obaidellah 2020; Al Madi et al. 2021; Fritz et al. 2014; Vrzakova et al. 2020], e.g., "predict where and when the eyes move across a line of code" [Al Madi et al. 2021] and detect "confusion and misinterpretation" [Vrzakova et al. 2020].
  
• Model validation and enhancement (3 papers): this action concerns employing another data set to test the validity of the trained model [Ahsan and Obaidellah 2020], and two ways of enhancing the models with enriched data: training data diversification (e.g., [Villamor and Rodrigo 2018]) and balancing [Abbad-Andaloussi et al. 2022]. One study may mention both enhancement methods. Both training data diversification and balancing increase the scale of the training data.
  
  • Training data diversification (3 papers): include new data from a completely new cohort or scenario [Abbad-Andaloussi et al. 2022; Ahsan and Obaidellah 2020], or with new features of existing data points [Villamor and Rodrigo 2018].
    
  • Training data balancing (1 paper): increase sampling of the current under-represented or hard-to-recruit participant cohorts [Abbad-Andaloussi et al. 2022].
    
• System improvement (3 papers): the envisioning of integrating the trained model into existing systems or tools to better support programmers [Fritz et al. 2014; Lee et al. 2018; Vrzakova et al. 2020].
  
• In-situ evaluation (3 papers): the intention to evaluate the trained model on data from a work environment [Hijazi et al. 2021; Vrzakova et al. 2020] or with a more realistic user study [Zhang et al. 2019].
  

The typical limitations of a mapping study [Budgen et al. 2018] relevant to the work presented here are the study selection process and the data extraction process. For the study selection process, the initial set of articles was based on a search string in an academic database. We chose a well-known database, which we believe is rather complete. The search string was iterated upon in order to find all relevant articles, but there may still be articles that we did not find with our search string. To mitigate this concern, we have included extra search terms to find relevant articles from the EMIP community, as described in Section 2. The selection of studies was further performed by consensus of multiple authors. Given these actions and our own knowledge of the field, we believe that we have identified a reasonably good set of articles. For the data extraction process, the extracted data was iteratively discussed within the author group, e.g., based on previous research and knowledge of the research field. Since we decided to collect rather well-known, accepted and explicit types of data, such as the type of machine learning model, we believe that no large errors or misunderstandings have occurred in the data extraction and analysis.