PD-Insighter: A Visual Analytics System to Monitor Daily Actions for Parkinson's Disease Treatment

Visual analytics tools have supported various clinical decision-making tasks, such as analyzing medical events over time [48, 69], determining demographic risk factors and related outcomes [54, 70], and monitoring longitudinal disease progression [59, 62]. Using such tools in conjunction with remote patient monitoring systems, health experts can observe patients away from the hospital using various health metrics, including blood pressure, pulse, heart rate, body temperature, and oxygen levels [15, 43, 50]. Remote monitoring approaches would significantly benefit people with PD, whose motor impairments can make clinical visits especially challenging [20]. Body-worn sensors can collect data about people's movement patterns for remote monitoring applications; however, data from such sensors tends to be insufficient for clinical analysis when used in isolation (see Del Din et al. [17] for a survey).

Activities of daily living (ADLs) can provide context for interpreting body-worn sensor data. ADLs describe the essential abilities needed to self-sufficiently manage one's care, including eating, showering, and walking. Medical specialists consider ADLs crucial in diagnosing and treating conditions such as PD [4, 16, 42]. People with PD may exhibit motor deficits differently across ADLs. For example, a pronounced forward lean is normal while reaching for an object in front of you but may indicate a postural impairment when walking.

Understanding activity data during ADLs often requires a knowledge of the patient's environment to discover how objects in the home might play a role in movement patterns. The presence of a couch and dining table correlate with sitting and low activity while transitions between spaces correlate with walking and high activity [71]. Past studies of ADLs found that medical specialists benefit from contextualizing behavioral data based on the patient's location (e.g., kitchen, bedroom, bathroom) [53]. Other systems visualize ADLs with graphical representations of the patient's apartment, using trajectories or avatar replays to show movement in space [9, 51]; however, these approaches only provide short snapshots with limited global insight into movements over time, requiring analysts to remember key patterns in ADL data throughout the replay. Gil et al. [26] summarized activity within the home using embedded sensors on common objects to infer the kinds of activities people engaged in over time. While this approach provides knowledge about the number of actions occurring in various locations, it provides limited insight into movement quality.

Other approaches enable clinicians to analyze specific biomechanical motions from a given activity in detail. Ploderer et al. [49] visualized upper limb movement in people who had a stroke using multiple representations of arm movement across time, the amount of time active, and heatmaps on 2D avatars showing body areas with high activity. HAExplorer [23] displays precise helical components of biomechanical motion at individual joints. Keefe et al. [33] provided a small multiples view of biomechanical motions from a specific activity to view from multiple perspectives. However, these approaches focus on specific motions and joints over discrete activities in isolation rather than global, multi-activity perspectives and environmental contexts associated with motor abilities in daily life. Other systems used Kinect for data capture to create a full-body skeletal reconstruction and extract kinematic features, capturing in home body motion data for clinical analysis and PD diagnosis [31, 47]. However, for ADLs, skeleton reconstructions and kinematic values can be difficult to analyse without environment, temporal, or action context.

Motion analysis for PD and similar motor disorders must balance a crucial tension between access to high-level summaries showing how body motions change over time (i.e., hours to days) and low-level details enabling deeply contextualized analysis over brief movements or activities (i.e., seconds to minutes). Systems representing ADLs with activity level or location can present data across long time periods but lack environmental and full body contexts to draw meaningful conclusions about behavioral patterns. Replays of patient motions can provide essential context but only work for brief time segments. We pair statistical, activity-based overviews with deeply contextualized immersive reconstructions to enable an understanding of large body motion patterns across time and the analysis of context.

To help clinicians analyze patient activities in context, PD-Insighter combines traditional visualization workflows with immersive analytics (IA). IA leverages immersive technologies such as virtual and augmented reality (VR and AR) for data visualization [44]. Immersive environments project data into 3D space, often making navigating, manipulating, and perceiving spatial information such as depth, height, and size more intuitive in part by integrating binocular cues and by allowing people to physically move through data [29, 36, 44, 63, 67]. Improved spatial perception from AR is especially important to our work, as analysing 3D body limbs in space requires accurate understanding of depth and distance. Our work is particularly informed by research in situated analytics (SA), which directly integrates visual representations of the data into the studied environment, often placing data on top of physical objects such as buildings or bodies [10, 65]. SA can incorporate data directly into the physical environment, advancing fields such as fieldwork [64], construction [30], and interior design [61].

Past work in SA summarizes human motion within an indoor environment, preserving the time, place, and interactions with the space [34, 41]. For example, AvatAR [52] paired a digital avatar with different embedded representations of motion (e.g., gaze, paths) to reflect movement patterns and interaction with the environment. However, these systems primarily visualize where people have interacted with the space rather than the quality of their movements. We instead focus our approach on understanding the mechanics of patient motion, where context and location play a vital secondary role in interpreting movement patterns.

SA systems have also situated visualizations with respect to the body, supporting more detailed mechanical motion analyses, such as improving cycling performance [32], basketball free throws [39], hand gestures [24], and physical rehabilitation [8, 21, 25]. Real-time systems such as Physio@Home [58] used AR mirrors to embed wedges and lines directly on the person's body to guide movement for upper limb physiotherapy. YouMove [7] also employed an AR mirror to teach physical movement sequences by having people follow along with a skeleton's actions. These systems demonstrate the power of IA for real-time motion analysis and guidance; however, they focus on short time-frames for guiding motion in real-time.

While traditional visual analytics approaches can support coarse-grained longitudinal analysis, past work in IA suggests its potential for contextualized investigations and replays of body motion and key actions. Clinicians can use IA tools to understand the full context of a movement and view body motion from multiple angles to gain a holistic picture of potential motor deficits. Further, IA can offer visceral experiences with data [37, 72], which may augment data interpretation by emphasizing relative proportions of the patient's physical space and associated objects and actions. We pair this paradigm with traditional overview analytics in PD-Insighter to enable multi-scale insight into movement data for physical therapy for PD.

People with PD develop various motor deficits, including gait disturbances and balance impairments that can lead to falls, gait freezes, and difficulties standing and grabbing an object [45]. To determine effective rehabilitative treatment methods, clinicians must efficiently and effectively identify motor deficits and their potential triggers. Triggers can include changes in the environment [5], time since medication [6], and divided attention [13].

Data for body motion during ADLs could provide clinicians with crucial information about deficits and triggers before meeting a patient. For example, if a person with PD expresses challenges standing up from their chair, a clinician could examine their data prior to a clinical visit to determine potential triggers causing this difficulty. The clinician could then spend the clinical visit educating the patient on exercises or compensatory methods to improve sit-to-stand transfers. Due to time constraints in the clinic, clinicians require rapid and efficient methods for reviewing this data.

To understand patient motor behavior, clinicians need to 1) focus on the patient's ability to perform specific ADLs (e.g. walking, standing, reaching), 2) discover when and how frequently motor deficits occur, and 3) understand movement context to see why motor deficits occur. While clinicians currently rely on brief in-clinic assessments and self-reports, data-driven approaches can provide more precise and holistic insight into patient movement.

While the terminology may vary across practices, clinicians identified three kinds of data that would support clinical analysis and decision-making for PD: body, action, and event. For this paper, body variables are specific joint angles and displacements where changes in value can indicate a patient's motor performance. An action is the process of moving one's body to achieve a goal, such as walking, sitting, or reaching. Actions can make up ADLs (e.g., reaching is performed while bathing). An event is the time period an action occurs.

Clinicians can infer motor deficits from shifts in body variables; however, clinicians can only fully make sense of body variable changes in the context of performed actions. For example, a patient may stop moving due to a freeze or due to external stimuli such as searching the refrigerator. Thus, clinicians desired a way to contextualize body variables with respect to action data, temporal events, and the physical environment to readily enable clinicians to investigate motor deficits and their potential triggers.

While clinicians do not currently have tools for working with longitudinal body motion and action data in PD treatment, we identified core tasks associated with such data through multiple clinician interviews. These tasks focused on data needs for understanding proper and improper body motion and what motion data indicates about the patient's condition. We revised these tasks through continued iteration with clinical collaborators on early system prototypes.

We characterize these tasks into three high-level goals: identify and filter data by action, where clinicians can assess movement patterns over key ADLs; discover motor deficits, where clinicians can compare motion patterns across key body relationships to discover systemic deficiencies; and contextualize motor deficits, where clinicians can view potential deficits in context to understand potential triggers and treatments.

3.2.1 Task 1: Identify and Filter Data by Action. Clinicians focus on specific actions to understand a patient's challenges. Actions consist of complex combinations of movements that together enable an intended goal. For example, the “walking” action combines synchronized and balanced footsteps, arm swings, and upright posture to move from one place to another. The patterns clinicians look for in motion data depend on the patient's action. For example, significant variations in trunk angle during walking may indicate a deficit, whereas the same variations are normal when standing up from a chair.

Figure 2 presents a detailed flowchart that outlines the sequential workflow of our system, transitioning from left to right across four key stages. 1) "Data Collection", which includes "External Cameras", "IMU Sensors", and "Smartphone Video". 2) "Data Processing", which includes "Action Labels", "Body Pose Estimation", and "Environment Mesh", and are derived from the collected data in the first stage. 3) "Design Integration", which includes "Overview Dashboard" and "Immersive Replay", both designed to visualize and interact with the processed data from stage 2. 4) "Deployment" with the dashboard on a "Laptop" and the replay on "HoloLens 2", illustrating the system's data journey from collection to application.

Key actions for informing treatment protocols include:

• Sit-to-stand (e.g., when a patient gets up from a chair)
  
• Sitting
  
• Stand-to-sit (e.g., when a patient sits down on a chair)
  
• Reaching (e.g., when a patient grabs something from a high shelf or across a table)
  
• Walking
  
• Standing
  
• Taking medicine
  

While other actions may also inform protocols, we focus on these common actions as our clinician collaborators identified these as necessary to understand a patient's condition. By letting clinicians examine motion patterns with respect to actions over time, clinicians can more readily identify and select time segments, filter and compare events of the same action, and select a specific event based on actions of interest (see §6).

We implemented the dashboard using D3.js and HTML5. We built the Immersive Replay in Unity and deployed it to a HoloLens 2 headset. We managed data processing and reconstruction using custom Python scripts, EasyMocap [2], and RealityCapture[1].

To develop PD-Insighter, we collected body motion data from a combination of RGB cameras and IMUs, following practices used in current clinical observations conducted in Parkinson's research and similar studies [11]. We conducted our data collection in a 11ft × 15ft environment, shown in Figure 3, which simulated a domestic living space (i.e., living room and kitchen) outfitted with everyday furniture items such as couches, counters, tables, chairs, and shelves and essential appliances including microwaves, coffee makers, utensils, and dishes. We captured the local environment using a smartphone video recording. The tracked person wore ten IMU sensors on their body to collect body pose data. Three large cameras on tripods record the person in the room performing actions of daily living. We processed the smartphone video into an environment mesh and the IMU data and camera recordings into a human body pose estimate, body variables, and action labels. We collected movements with acted-out PD behaviour from two members of the research team (one developer and one clinician) and real movements from a person with PD.

To support identification and filtering by action (Task 1), selecting an action enables clinicians to focus their attention and analysis on particular actions of interest. Predefined buttons display time periods when the patient performed the selected action (Figure 5.1). Our initial prototype presented all motion data over time with accompanying actions, using only temporal sliders for navigation. However, clinicians felt this presentation was overwhelming and preferred to use actions to isolate time segments of interest.

PD-Insighter represents each action event with a horizontal bar. The bar's width is proportional to the duration of the event, preserving temporal ordering and showing which events are shorter or longer. We map each action to a distinct color in a categorical palette from Color Brewer [28] to provide a compact action representation on the Timeline window. Each action type is presented in its own stacked row so clinicians can view simultaneous actions. For example, a high trunk angle while standing might be concerning to a clinician, but a high trunk angle while standing and reaching would not be concerning since the reach would explain the presence of the increased trunk lean (Figure 5.3a).

Figure 6.1 shows a chi-square line graph distribution overlaid on a red color ramp. A slider is under the graph and color ramp. Besides the slider is a red heatmap. As the slider moves to the left getting closer to the hump of the curve, the heatmap's values gets darker. Figure 6.2 shows two line graphs. The line graph labeled "normal" shows a gaussian curve. The second line graph labeled "abnormal" shows a curve heavily skewed to the right. Figure 6.3 shows two temporal heatmaps. The heatmap labeled "default" shows a lot of color with light and dark values. The heatmap labeled "simplified" is a simplified version of the "default" heatmap, showing much less variation with bins of white, green, and dark green values.

To help clinicians discover motor deficits (Task 2), the Body Variable Controls allow clinicians to display relevant motion data through body variable buttons and sliders (Figure 5.2). Clinicians use body data to efficiently compare the action events and determine body motion patterns, potential deficits, outliers, and specific events for further study. We display our body variables—trunk angle, arm use, foot position, and weight shift—with four temporal color-encoded heatmaps to provide a compact representation of the data over time and to enable clinicians to quickly spot outliers. We assign each body variable a color ramp from Color Crafter [56] using distinct hues for each variable. Body variable color legend sliders allow the clinician to change the maximum value of a color ramp to make desired value ranges more salient.

Overlaid on corresponding color legends, the line graph distributions for each variable show the frequency of the body variables across the dataset. These distributions allow clinicians to more rapidly tailor the color encoding in the Timeline to specific value ranges (Figure 6.1). Given the importance of left-right relationships in the feet and hands (see §3), we pair their distributions, distinguishing the left from the right using a dotted line. Distributions can also summarize balance and posture for selected time segments to assist in global motion analysis. For example, when a patient has a balanced weight shift and upright posture, the distributions show a Gaussian curve and equivalent left and right measures. Unexpected distributions, such as an uncentered weight shift curve, high trunk angle values, or unmatching left and right distributions, can reflect poor balance, posture, or gait (Figure 6.2).

Figure 7.1 displays a bar chart reflecting the amount of time the patient spent performing each action. Beneath the bar chart, a colored horizontal bar indicates the timeline of when actions were performed during the capture period. Figure 7.2 depicts ’Walking Events’ with three colorful temporal heatmaps for three walking events. Light blue bars represent times when PD-Insighter predicts a freeze during a walking event. Figure 7.3 shows ’Event Data’, featuring one of the heatmaps from Figure 7.2 enlarged, along with two range sliders and ’Play’ and ’Pause’ buttons.

The Timeline displays detailed body and action data over time based on the clinician's selections using a series of linear heatmaps (Figure 7). When no actions are selected, the clinician sees a concise synopsis of the patient's behaviors and movements throughout the dataset (Figure 7.1). The Action Summary uses a bar chart to summarize the time spent in each action and reflect the patient's general activity levels. For example, if the clinician sees the patient sitting most of the time, they may suggest that the patient walks around and stands more frequently. The Action Timeline uses a color-coded timeline to visualize the patient's actions chronologically, reflecting when and how long potential events of interest took place. This timeline helps clinicians understand sequences of related actions as well as general patterns of behavior, such as when the patient took their medicine, how long they were sitting, and if they were walking for longer or shorter periods of time.

Clinicians begin their analysis by selecting an action from the Select Action window or Action Summary chart. The Timeline window then updates to display a set of stacked temporal heatmaps reflecting the distribution of each body variable over time, with whitespace between discrete action events (Figure 7.2). Additional temporal and body filters (e.g., More than 5 Seconds, Imbalanced Weight Shift) shown in Figure 5.3b allow the clinician to filter even further and center their attention on events that fit their analysis goals. For example, the "Potential Freezes" filter was developed from mathematical calculations using the foot position, described in Section 5.2.2. Using actions and body variables to drive data filtering allows clinicians to focus on semantically meaningful sets of movements.

The large number of frames in a dataset can introduce clutter and make it difficult for clinicians to find outliers and patterns in the data. PD-Insighter provides a toggle in the Body Variable Controls to simplify the display to emphasize large shifts in motion data. Based on prior work in large-scale data aggregation [3, 14], our simplification feature bins data into mean bins for values within one standard deviation from the mean, outlier bins for values more than one standard deviation from the mean, and averages the values in each bin (Figure 6.3). By separating the data into these bins, we preserve and emphasize relevant outliers.

Once the clinician finds an event of interest, they can click on that action event to focus exclusively on data for that event (Figure 7.3). Two range sliders—a global slider with time relative to the entire capture and a local slider with time relative to the specified event—control the time segment in the temporal heatmap. The global slider can expand the Timeline view to show what happened before or after the selected time segment, as well as show where the selected time segment is with respect to the entire dataset. The local slider enables more precise navigation for zooming into the data. The slider also presents the start and end time to contextualize when in the day this happened. Once a clinician finds a moment they want to study further, the clinician can use the play/pause buttons or play slider to fetch video and reconstruction data to play in the Detailed View or Immersive Replay to explore an event in context (Task 3).

Figure 8 shows a subsection of the Dashboard, with two sections shown. 1) Underneath a red temporal heatmap labeled trunk, there is a horizontal orange bar labeled "Walking", two range sliders, and two buttons "Play" and "Pause". 2) "Detailed View" showing "Duration of Time", "Weight Shift: Right", and "Video" with an image of a person unbalanced.

The Detailed View presents statistical metrics and video replays for selected data. Based on clinician feedback, weight shift balance is a critical measure that can indicate when patients favor a particular side and potentially dangerous imbalances that can lead to a fall. The Weight Shift metric provides an average weight shift side, weight shift measures, and a text-based description of the weight shift. While access to specific measures provides precise comparisons, clinicians indicated that the raw measures could be difficult to interpret. We added text descriptions (e.g., "strong right," "slight left," or "balanced") to support this analysis.

Some metrics are more useful based on the selected time segment(s), so the Detailed View displays different metrics based on the current selection. For example, when no action is selected, clinicians can look at the overall percentage of time sitting, which is useful since large periods of inactivity can encourage motor decline. When the clinician selects an action, the Duration of Time metric shows how much time was spent performing that action, which can shape interpretation. For example, two freezes in one hour is more concerning than two freezes across two days.

When the clinician selects an action event to closely study, the Detailed View allows clinicians to replay 2D video data (Figure 8.2). Video data from our data capture provides ground truth to compare with the temporal heatmap to confirm if outliers are from motor deficits or errors in data collection or processing. Further, the 2D video segment can provide some context for interpreting motion variables, supporting Task 3. However, clinicians felt that the video replays only offered limited perspective into body movement. To support more detailed replay, clinicians can launch the Immersive Replay to explore the target movements.

Figure 9 shows an individual wearing an augmented reality headset. This person observes a digitally-rendered skeleton within a virtual environment, simultaneously managing the Overview Dashboard on a laptop.

The patient's body is reconstructed using a simple skeleton. We use this design rather than a more photorealistic reconstruction because it is visually simpler, provides additional patient privacy, decreases occlusions, enables situated visualizations of relevant body metrics (e.g., motion vectors), and requires less memory. We colored the skeleton bright red to distinguish the body from most naturally-occurring environments.

We visualize body variables using arrows at relevant coordinates to show the associated displacements in physical space. Arrow magnitude and direction correspond to the magnitude and direction of the body variable at that frame. The colors of the arrows match the colors of the body variables in the Overview Dashboard to connect the Immersive Replay with the studied body variables.

We situate the skeleton within an immersive reconstruction of the environment extracted from video data (see §5.2.1). PD-Insighter centers the reconstruction on the skeleton, showing a 1.5 meter radius window of the environment around the body. Original iterations of the Immersive Replay involved the body reconstruction moving in a stationary environment. However, clinicians noted that any analysis would likely happen in their office before a patient visit. The clutter and limited space of the clinician's office inhibit walking around large reconstructed environments, whereas centering on a smaller local window means clinicians can more easily walk around the local reconstructed environment to study contextualized motion patterns in detail.

PD-Insighter uses the HoloLens 2 headset to enable clinicians to seamlessly transition between the Overview Dashboard and Immersive Replay by shifting their gaze between the desktop display and the immersive reconstruction. For easier mobility, clinicians can view the Dashboard on a laptop and physically move around the immersive reconstruction to look at the patient's movements from multiple views. Early iterations of the system used Oculus Quest 2 to display both the dashboard and reconstruction, but clinicians found using the device controls for interaction cumbersome. We switched to see-through AR to allow clinicians to use the desktop to interact with the Overview Dashboard and provide more comfortable physical interaction with the Immersive Replay.

All participants engaged with the opening Action Summary and Timeline (Figure 7.1), which C1 described as “necessary and grounding,” and would frequently return to after studying an event. C2 and P3 used the Action Summary bar chart and the Percentage of Time Sitting metric to see if the patient was active or inactive. P6 thought the Weight Shift statistic would be helpful because “if you don't have time, you can really quickly see a snapshot that tells how [the patient] has been doing." P3 remarked that the feature would be helpful for clinical analysis since “the amount of sitting tells a lot about how active someone was, which is helpful even for stroke patients, not just Parkinson's.”

Participants also focused on using the body variable distributions to identify patterns in motion across actions. C1 and C2 were interested in whether the patient used both arms or just one arm while reaching because “Typically in Parkinson's, there can be an asymmetric representation [between arms] because they might be compensating. [For example,] if there is a tremor, they could be hesitating to use that hand” (C1). By studying the Arm Use heatmaps for reaching events, C1 and C2 both found that there were several reaching events with single dominant hand and bi-manual reaching, indicating “no concern” (C1; Figure 10.1).

The Action Timeline was “useful for thinking of the context of what [the patient] was doing through the day. When are they taking medicine? Are they walking for shorter or longer?” (C1). P4 also remarked “It is helpful to know at what point they are taking medicine because it has a huge impact on their motor function. It takes about an hour for medicine to kick in.”

Figure 10.1 displays green heatmaps labeled ’arm left’ and ’arm right’ during reaching events. Some heatmaps show dark green for both the left and right arms, while others show dark green bands only for the left arm. Figure 10.2 depicts red heatmaps labeled ’trunk’ during walking events, with an arrow pointing to one of the heatmaps that features a dark red area. Figure 10.3 presents green heatmaps labeled ’arm left’ and ’arm right’ during walking events. Arrows point to two heatmaps, where the right arm's heatmap is nearly white, and the left arm's heatmap shows dark green bands. Figure 10.4 illustrates red heatmaps labeled ’trunk’ during sit-to-stand events, with an arrow pointing to the longest heatmap that contains alternating dark red and light red bands.

All participants used the Select Action feature when initiating a new line of inquiry. Once participants picked an action, they chose body variables of interest to display. Participants frequently remarked how various actions impacted their expectations for the body variables. For example, “high trunk is concerning for walking, but not for reaching” (C2), and “Arm use while walking will be different than arm use while reaching” (C1).

Participants commonly focused on events with dark-colored outliers to find specific instances of motor deficits. For example, C1, C2, P3, and P6 were interested in moments with imbalanced arm swing while walking since “swinging with the left arm and not the right arm is characteristic of Parkinson's” (C1). When they selected Walking, Arm Use, and the Imbalanced Arm Use filters, participants were interested in instances where the right arm heatmap was close to white, and the left arm heatmap had dark green bands (Figure 10.3), reflecting extended arm swing. Participants felt confident that imbalanced swings were occurring without viewing a replay.

P3, P4, and P5 clicked on a Walking event with dark red bands indicating a high trunk angle and found a freeze (Figure 10.2) missed by the Potential Freeze filter. C2 clicked on a Standing event with a dark red band and found a fall. P6 clicked on a Sit-to-Stand event with dark a red band because “a lot of falls are happening from leaning too far forward.” C1 clicked on a Sit-to-Stand event with multiple dark red bands, which reflected an event where the patient was trying to lift their weight out of the chair and struggling (Figure 10.4). P3 and P4 used the blue heatmaps to determine weight shift imbalances. P4 remarked that “in general, the weight shift is more on the right than the left,” and was interested in this imbalance during walking events because it showed the patient “having a prolonged period of single limb stance.”

P5’s navigation was heavily driven by color contrast to find potential deficits, “Really what I am looking for, just for the sake of it, is a moment where one (heatmap) is darker than the other (heatmap).” When looking at Stand-to-Sit events, P5 had all body variables selected, saying, “I am going to click all of them until I can decide what I want to look for,” and chose the event with the darkest trunk angle. Then, P5 looked at the Foot Position and chose the event with the darkest purple. This behavior indicates PD-Insighter's ability to support data analysis based on unexpected and significant shifts in different parts of the body relative to others, highlighting outliers and potential deficits.

Participants also used time duration to find moments of interest. For example, C2 wanted to see if the patient abruptly fell in their seat, indicating a “failure to react to a change in position.” C2 selected the Stand-to-Sit event with the shortest width and found the patient fell into their seat. P3 and P4 were interested in long Sit-to-Stand events indicated struggles standing up. P3 clicked on the More Than 5 Seconds filter to find longer events, whereas P4 looked at the width of the bars to “see if they are getting better (at sitting down) over time.” Both participants clicked the same long Sit-to-Stand event that showed a struggle to stand up (Figure 10.4).

Filters helped decrease the number of events shown and focus participants’ explorations. All therapists clicked on the Walking and Potential Freezes filters to try to find freezing gait patterns. PD-Insighter showed three different walking events, with light blue bars indicating potential freeze moments (Figure 7.2). By using the immersive replay to watch the reconstruction and see if there were body leans showing the patient trying to start walking but struggling, it took clinicians less than one minute to find that one of these events was a proper freeze.

Figure 11.1 shows a red skeleton with its arm extended backwards, and a green arrow extending from the hand, pointing forward. Figure 11.2 shows a red skeleton with a blue arrow extending from the pelvis, pointing to the left. Figure 11.3 shows a red skeleton with two blue arrows extending from the pelvis; the one pointing to the left is much shorter than the one pointing to the right. Figure 11.4 shows a red skeleton with its hand on a virtual table. Beneath all these figures are colored heatmaps corresponding to the magnitude of the arrows over time.

While participants had access to the ground-truth video feed in the Detailed View, they primarily used the Immersive Replay for watching body motion because they felt “the video is too constrained” (P3) due to the inability to move around the body and look around any occlusions. The replay in AR was helpful because “angles matter, I need to orient myself to see [body parts of interest]” (C1). All participants found the headsets easy to use with “no discomfort” (P6).

Therapists used the Immersive Replay to confirm or deny the existence of a motor deficit in an event of interest. For example, when studying standing events, C2 saw a dark red band while reaching and “suspected it was just a normal reach since when reaching a high trunk angle is common.” However, the red was significantly darker than the other trunk leans while reaching, leading C2 to further investigate. When watching the reconstruction, they found the reach led to a fall. The reconstruction “helped me see how hunched they were, that it was a fall and not just a reach.” While looking for freezing events, all participants used the reconstruction to determine if a freeze occurred. When they saw the skeleton leaning back and forth and shuffling before stopping, they accurately characterized the movement as a freeze. Once discovering the freeze, C1, C2, and P4 returned to the Overview Dashboard to study the event in more detail. For example, C1 wanted “to see if the trunk increased, which is indicative of more risk for falls.” In the Overview Dashboard, they looked at the trunk angle heatmap and found that the trunk “looked higher but not substantially” with respect to the rest of the data.

Participants felt that the arrows encoding body variables in the Immersive Replay provided helpful clarity. When watching an event with an imbalanced arm swing, C2 said, “The arrows give help with the direction. I can see the arm swing is more backwards than forwards.” P4 said that “the arrows make it clear that the arms are obviously not synchronized” which leads to checking, “Are they using their assistive device or furniture?” (Figure 11.1).

Therapists also used the blue weight shift arrows for further analysis. For instance, “When [the patient is] turning while walking, I can see how the arrow changes to emphasize the weight bearing, and I can see more on one particular side.” (P3) (Figure 11.3). P4 liked this visibility because “turning is a time when people will fall. It's a good thing [for the patient] to shift their weight while turning.” P4 used the weight shift arrows to study a freeze event to realize that the patient was “trying to move one side but can't initiate” (Figure 11.2). After finding this imbalance in the arrows, P4 went back to the heatmap and saw that during the freeze event, the right side was dark blue, and the left side was white, confirming that the patient was placing their weight on their right side to try to move forward.

P5 used the weight shift arrows to study a Sit-to-Stand event, saying, “There's some bias on the right side, so [the patient] will do something like this when they are trying to get up,” and acted out a swivel to demonstrate the incorrect way to stand up. P5 considered the advice they would give the patient “Are they building strength on that [less affected] side? Maybe we need to adjust that chair, add pillows, wedge a book under the rocking chair.”

The environment provided participants with important context for analysis. When studying a Walking event, C1, C2, P3, P4, and P6 were curious about the high trunk leans in the heatmap. When watching the reconstruction in the Immersive Replay, C2 said, “This shows me that they are furniture walking [walking while using furniture as an assistive aid]. Before, in the heatmap, it looked like they were walking weird. I would have thought they were about to fall” (Figure 11.4). P4 remarked, “Yeah, that's definitely helpful to have the environment there. Otherwise, it would be like, what are the arms doing so far apart here?” when indicating a point of divergence between hand positions on the heatmap. C2 mentioned the importance of seeing the environment and furniture walking, “When a patient knows their center of balance will be disturbed, I want to see interaction with the furniture to help stabilize.”

Most ADL visualizations represent data through activity level and location, which does not give the context necessary for identifying and studying PD motor deficits. We discovered that combining action and body variables can indicate motor deficits and body pattern changes. Clinicians noted that the action and body variables were necessary for a holistic picture of patient movement.

Based on the range of patient needs, we developed action labels and body variables from existing clinical practices to assess patients. P6 appreciated the chosen action labels, "These features are known from clinical observation research to be associated for risk of fall." Different clinicians can use the tool based on their expertise or the needs of their patients. P4 stated, “I like having independence to look at what I want to look at based on what is important to me,” which can be “influenced by patient's needs and what they are reporting.” P6 described how actions of interest can also be influenced by personal specialty, “I am clinically more interested in walking, sit-to-stand, and stand-to-sit. How someone is sitting or standing is less relevant to me.”

Our evaluation showed that action labels and body variables successfully indicated motor deficits such as freeze events, imbalanced arm swing while walking, struggle standing up, and struggle sitting down. Clinicians successfully understood body motion in terms of these new body variables and could identify when the data was reflecting improper motion. While various data representations exist for body motion, both action and body data can work together to enable scalability across time and provide the necessary context for deeper analysis.

Most moments of clinical interest for PD body movement last only a few seconds. Finding these moments in an hour or more of data is time cost-prohibitive with direct replay: clinicians only have a few minutes to prepare for a patient visit. Emphasizing outliers in the data using dark, saturated colors helped quickly draw clinicians’ attention to potential deficits.

At the beginning of the evaluation, P3 thought that the color encodings would be challenging for PTs unfamiliar with this representation. However, within a few minutes, they found that “Using [the colors] gets intuitive very quickly” and had no issue interpreting the data. P3 described the heatmap as “very telling. I don't need additional information to see the weight shift balance, a quick glance tells which side bears more weight.” Participants found that the simplification feature made the heatmaps easier to use (P3, P4, P6). C2 found the body variable sliders helpful to “adjust the contrast [so that] it's easier to see” and the distributions helpful to “guide where to move the sliders.”

Task-aware aggregation methods, such as the simplify feature, can highlight these outliers without removing important information, which is helpful for large and noisy data. While several approaches for aggregation exist, we found that clinicians needed methods that both preserves and emphasizes outliers to ensure that important moments of motor deficit are not lost.

When we developed PD-Insighter, we used datasets with manual labels. To date no tools exist for automatically and reliably labeling the specific activities of interest and we believed that accurate action labeling was critical. While accuracy is desirable, we also discovered that PD-Insighter created a certain degree of fault tolerance in motion analyses. Actions or relationships between body variables support quick, coarse guidance toward events of interest, and verifying events is sufficiently quick that clinicians could readily confirm or dismiss events. For example, although our Freeze Event thresholds introduced several false positives, clinicians did not mind the inaccuracies because they could confirm the event in a few seconds by simply watching a replay.

Salient misalignments in body variables helped clinicians capture significant instances that action labels may have missed. For example, participants quickly began looking for time spans in the heatmaps where body variables were high in one value and low in others, helping address potential false negatives in labeling. As clinicians worked with the tool, they generated heuristics, such as those for the Freeze Event filters, that gave coarse approximations of critical potential deficits and informed clinical thinking about the computational relationships between joints in different movements. This introspection may enable future metric development and on-the-fly filtering for working with biomechanical data.

The combined fault tolerance and developed mathematical intuitions about deficits exhibited in the evaluation indicate that approaches like PD-Insighter may support mixed-initiative action classification. Specifically, if an automated classifier embedded in PD-Insighter provides preliminary action labels, clinicians may confirm or deny events through their analysis. This confirmation can provide clinicians with valuable patient motion insights and classifiers with additional training data. Improving overall classification feedback and enabling customization for individual movement patterns has important potential for future research. Exploring the potential for such a system is important future work.

While the abstract heatmaps helped clinicians discover that an unexpected motor pattern occurred, they typically cannot explain why the observed body motion change happened. Replays helped clinicians distinguish between motor deficits, inaccurate measures or heuristics, and potential triggers. All participants felt the video replay was inadequate for this analysis and preferred the up-close immersive reconstruction. Five of the participants walked around the model at least once during the analysis to look at the body from multiple angles. C2 emphasized the importance of having access to various perspectives by pointing out “It's hard to see someone leaning forward from a front view; I would want a side view, and here I can.” Clinicians study a patient's body motion in person, and AR recreated familiar experiences for retrospective analysis and delivered a better sense of the scale of the movements.

The body and environment reconstruction work together to provide the necessary context. AR provides crucial depth and scale to analyze body motion properly. Further, AR provides necessary contextual cues from the environment. Reflecting on a prior iteration of the system without the reconstruction, our collaborator C1 noted the replay was “much more helpful with the environment. I can see what they are doing and why they are doing it.” Clinicians used the replay to analyze object placement in reaching, walking with furniture assistance, and chair placement when sitting.

The functional gains from combining both interfaces outweighed the trade-offs associated with context switching between platforms. While clinicians found video replays in the dashboard inadequate, early iterations of the system experimented with situating the Overview Dashboard in a full VR reconstruction. However, clinicians had difficulty interacting with the data. With AR, no participants noted difficulties transitioning between the Overview and Replay. C1 described the interface as “fairly seamless”; the reconstruction is “right there.” Despite having no prior experience with head-mounted displays, P6 described the AR headset as "good, awesome, better than I expected," because "it does not prevent you from seeing other areas in the real world."

With participants finding the headsets easy to wear and interact with, as well as useful for viewing the replay in multiple perspectives, our findings align with recent research on how immersive renderings can improve 3D data analysis [39], emphasizing the importance of comprehending trade-offs in visualization displays for specific design scenarios [22].