Reading a Remote Teammate's Body: A Body-Worn Sensing Interface for Group Coordination in a Distributed Interactive Space
DOI: https://doi.org/10.1145/3845796.3859311
ISS Companion '26: Companion Proceedings of the Conference on Interactive Surfaces and Spaces, Turin, Italy, November 2026
Teammates who share a room read one another's effort and strain from subtle bodily cues, many of which are lost in distributed interaction. We present a body-worn sensing interface that renders a remote teammate's heart rate (HR) and heart rate variability (HRV) as a single animated halo in a game's heads-up display. Pulse, brightness, and expansion encode HR, while ring coherence encodes HRV; states are classified relative to each player's baseline and carry no emotion label. In Study 1 (N = 36), participants identified HR states on 66.3% and HRV states on 58.2% of trials, both above 25% chance, and identified both jointly on 44.6% of trials against 6.25% chance. Study 2 evaluated the same encoding with an independent sample of 12 three-player teams using live Apple Watch sensing under No Cue, Numeric Cue, and Icon Cue conditions. Icon Cue reduced support response time by 3.90 s relative to No Cue and showed no statistically significant difference from Numeric Cue in teammate awareness, with lower workload and perceived invasiveness. Numeric Cue, however, scored higher on coordination usefulness and trust. Together, the studies show how an ambient physiological representation can support remote coordination while preserving a meaningful trade-off between subtle disclosure and explicit physiological values.
ACM Reference Format:
Romman Jahandideh, Aran Jahandideh, and Steve DiPaola. 2026. Reading a Remote Teammate's Body: A Body-Worn Sensing Interface for Group Coordination in a Distributed Interactive Space. In Companion Proceedings of the Conference on Interactive Surfaces and Spaces (ISS Companion '26), November 23–26, 2026, Turin, Italy. ACM, New York, NY, USA 7 Pages. https://doi.org/10.1145/3845796.3859311
1 Introduction
In a shared room, teammates register a partner's quickened breath and rising tension continuously and prereflectively. Distributed interaction removes much of this layer: remote partners infer effort from text, voice, and avatar motion, and removing face-to-face cues makes collaborators judge one another less accurately [15]. Two recurring approaches in prior work are affective inference from sensed signals and explicit physiological sharing. Affective computing established a tradition of inferring affective states from sensed signals [14], while physiological-sharing systems have explored how biosignals can support awareness and social connection [9]. Howell et al.’s ambiguous skin-conductance display withholds fixed interpretive labels [10]; this work adds a separately encoded second channel and a chance-referenced legibility test.
We ask: can a remote teammate's physiological state remain readable, and useful for coordination, without converting the body into an explicit emotional or diagnostic label? This work contributes to ISS at the intersection of body-worn sensing, peripheral displays, distributed interactive spaces, and group coordination. We contribute: (1) a design approach that translates Science and Technology Studies (STS) commitments into a compositional two-channel HR/HRV encoding; (2) chance-referenced evidence from Study 1 that the label-free representation is legible and graded; and (3) a live multiplayer evaluation showing that the Icon Cue accelerates teammate support relative to No Cue and shows no statistically significant difference from Numeric Cue on teammate awareness, with lower workload and perceived invasiveness, while Numeric Cue remains higher on coordination usefulness and trust.
2 Design: A Trace, Not a Verdict
Classifications that persist and travel accumulate consequences for the people they sort [2], measurement is not a neutral window onto a preexisting state [1], and any reading of a body is situated and partial [7]. The display therefore offers a trace to interpret in context rather than a verdict to accept. Each teammate is one animated amber halo (Figure 2). Heart rate, an index of momentary activation, controls pulse speed, glow, and expansion; heart rate variability, a marker of autonomic regulation [16], controls ring coherence, from a smooth halo to a broken, jittering one. Each channel has four states relative to the player's own baseline, from HR1 (within + 5 bpm or 10%) to HR4 (more than + 30 bpm or 45% above), and from V1 ($\ge 110\%$ of baseline HRV) to V4 (below 60%), giving 16 compositional states. These are prototype design states, not medical, clinical, or emotional diagnoses, and because they are baseline-relative they are not comparable across people. Hue stays a warm amber rather than an alarm red, and every state carries a small ambient shimmer so that the calmest state does not read as a flawless signal.
3 Study 1: Is the Halo Legible?
Thirty-six adults took part online in a within-subjects study (Figure 3). In an open phase, they described four unlabeled states. In a structured phase, they viewed all 16 states in random order, identified the HR and HRV state from four options each, rated clarity, urgency, stability, coordination usefulness, and support intention, and completed the Self-Assessment Manikin (SAM) [3]; three NASA-TLX items [8] closed the session. This is a recognition task: viewers judged isolated halo stimuli whose intended states were known.
Analysis. Following the ISO 9186 convention [11], participant-level accuracy was compared with chance using two-sided one-sample t tests, with Cohen's dz. All four pre-specified legibility comparisons also survived a conservative Bonferroni threshold of α = .0125. Planned linear trends were Holm-corrected across five tests. Free-text interpretations and open responses were analyzed using reflexive thematic analysis [4].
Results. Participants identified the HR state on 66.3% of trials (95% CI 58.5–74.2) and the HRV state on 58.2% (95% CI 51.0–65.3), both above 25% chance (p < .001). Joint identification reached 44.6% against 6.25% chance (t(35) = 9.90, p < .001), and the hardest joint cell reached 33.3%, more than five times chance (Figure 4). Holm-corrected linear trends showed that higher HR raised perceived urgency and arousal, and that a more fragmented ring lowered perceived stability and raised support intention by 0.48 points per HRV level (t(35) = 7.78, p < .001). Clarity averaged 5.22/7 and coordination usefulness 5.45/7; mental demand (3.64), effort (3.47), and frustration (2.53) remained below the scale midpoint.
4 Study 2: Live Multiplayer Evaluation
Method. An independent sample of 36 participants formed 12 teams of three (Prep, Cook, Runner) and played a custom Unity cooperative-cooking task modeled on Overcooked [5] from separate locations (Figure 5). Each player wore an Apple Watch; a HealthKit companion app exposed HR and beat-to-beat data, and a local bridge classified each player's HR and RMSSD-based HRV state against a three-minute seated baseline [12]. Only the resulting classified HR/HRV state values were replicated to remote clients; raw physiological time series remained local. Each team played three seven-minute missions, one per condition (No Cue; Numeric Cue, HR in bpm and HRV in ms; Icon Cue, the halo), Latin-square counterbalanced after a matched tutorial. Each mission included three researcher-triggered task events: Ingredient Bottleneck near 2:00, when two orders required the same ingredient; Pot Overload near 4:30, when two pots became active close together; and Service Rush in the final phase, when completed dishes, dirty plates, and delivery tasks accumulated. These events targeted Prep, Cook, and Runner, respectively. Physiological input remained live; the task events were scripted. Questionnaires followed each condition.
4.1 Measures and Analysis
Support response time was measured in seconds from actual pressure-event onset to the first relevant support action by another teammate. Appropriate support actions were coded separately: a support action was appropriate when it matched the immediate task need, for example passing ingredients, covering a pot, plating a dish, or delivering an order. Appropriateness was contextual rather than an intrinsic property of an action, so response time marks when support began, whereas the appropriate-support count captures whether support fit the need. Event logs recorded action type, appropriateness, prior verbal requests, and cue references. Service Rush also assessed whether appropriate support occurred within ten seconds. Recovery time was measured in seconds from actual pressure-event onset until the task bottleneck was resolved: the required ingredients had been prepared and supplied for both affected orders (Ingredient Bottleneck), both affected pots had been cleared (Pot Overload), or the dishes, dirty plates, and deliveries pending at event onset had been processed (Service Rush). Successful dish retrieval and burning were distinguished; newly arriving work was excluded from the Service Rush endpoint. Unresolved events were recorded as unrecovered. This measure describes task recovery, not physiological recovery.
Behavioral outcomes were analyzed at the team-by-condition level (n = 12 teams), using repeated-measures ANOVA and planned two-sided paired t contrasts (Icon versus No Cue, Icon versus Numeric, and Numeric versus No Cue), with Cohen's dz. The primary contrast was Icon versus No Cue for support response time. The 108 pressure events were nested within 12 teams, not treated as independent observations. Questionnaire comparisons used participant-by-condition paired t tests (n = 36 participants) over the same three contrasts. Analyses used Python with pandas, SciPy, and statsmodels.
Team communication was coded post hoc into eight categories: task instructions, support offers, status checks, uncertainty statements, help requests, strategy changes, physiological references, and numeric references. Coding relied on one primary coder without a verified reliability sample. Category frequencies were compared at the team-by-condition level with Friedman tests followed by Wilcoxon signed-rank pairwise contrasts. Answers to the closing open-ended questions were coded with theme labels applied during data collection; this is distinct from Study 1’s reflexive thematic analysis.
4.2 Results
Display reliability. All 36 participants completed all missions (108 pressure events). Latency averaged 180–247 ms, missing data 3.5–4.5% of samples, and displayed states matched classified states on all 555 logged samples from the cue-present conditions. This agreement establishes display fidelity, not observer comprehension (Section 5).
Support behavior. Support response time was 15.3 s under No Cue, 12.5 s under Numeric Cue, and 11.4 s under Icon Cue (Figure 6 a). Icon Cue was 3.90 s faster than No Cue (t(11) = −5.05, p < .001, dz = −1.46), and Numeric Cue was 2.87 s faster than No Cue (t(11) = −4.49, p = .001, dz = −1.30). Icon Cue was numerically faster than Numeric Cue (11.4 vs. 12.5 s), but this difference was not statistically significant (t(11) = −1.81, p = .098, dz = −0.52). Task recovery time decreased from 29.0 s under No Cue to 23.2 s under Icon Cue (t(11) = −5.31, p < .001, dz = −1.53). Teams logged more appropriate support actions under Numeric Cue (3.17) and Icon Cue (3.42) than under No Cue (2.25; descriptive team means, Figure 6 d). Uncertainty statements and help requests decreased when a cue was available, while support offers and strategy-change talk rose specifically under Icon Cue (Friedman tests, all p < .001; corresponding Wilcoxon contrasts, all p ≤ .002).
Awareness, workload, and perceived invasiveness. Teammate awareness rose from 4.21 under No Cue to 5.34 under Icon Cue (p < .001), with no statistically significant difference from Numeric Cue (5.42; p = .49). Workload was lower under Icon Cue than Numeric Cue (four-item descriptive mean 3.55 vs. 3.85, p = .004). The four-item workload mean had low internal consistency (Cronbach's α = .13) and should be interpreted descriptively rather than as a validated composite. Perceived invasiveness was lower under Icon Cue than Numeric Cue (3.19 vs. 4.39, p < .001), though higher than under No Cue (2.11), and comfort sharing was higher (4.81 vs. 4.33, p = .013). Numeric Cue was rated higher on coordination usefulness (5.50 vs. 5.30, p = .023) and trust (5.55 vs. 5.24, p = .002) (Figure 6 b).
Final preferences. Final preferences revealed a complementary pattern: 50% selected the Icon Cue as most useful and 53% preferred it for real cooperative play, whereas 47% selected the Numeric Cue as easiest to understand and 47% as most trustworthy; 78% selected No Cue as least invasive (Figure 6 c). Participants therefore distinguished between an ambient cue they preferred in play and exact values they trusted for interpretation: “The icon felt more like part of the game; the numbers felt like a measurement tool. The trade-off was speed versus precision” (P042).
5 Discussion and Limitations
Study 1 shows that the label-free encoding is readable well above chance; Study 2 shows that the same encoding can influence team behavior during live cooperative play. No statistically significant difference in teammate awareness was detected between Icon and Numeric Cue. Icon Cue had lower reported workload and perceived invasiveness, but Numeric Cue scored higher on coordination usefulness and trust. The result is therefore not a clean advantage for ambiguity. Instead, it suggests a hybrid design in which an ambient cue remains the default and exact values are surfaced when greater specificity is useful. We frame this as a design implication of these findings: default to ambience, and reveal precision when needed. Participants repeatedly asked for optional, mutable visibility, motivating opt-in sharing, an immediate off switch, and separate permissions for ambient icons and exact numeric values. Because each halo is driven by only two classified values, the same architecture could also support shared displays, tabletops, or spectator walls [13]; whether bystanders read the cue differently from teammates remains untested.
What the scripted events do and do not show. The two studies support different claims. Study 1 tested recognition: whether viewers could identify the intended HR and HRV state of isolated halos. In Study 2, physiological input was live, but the task-pressure events were scripted; event onset is the behavioral timing reference and does not establish a particular physiological response in the overloaded player. Agreement between displayed and classified states shows that the halo rendered what the classifier produced. It does not show that teammates understood the intended cue information during a given event: Study 2 included no event-specific comprehension check. Faster assistance under a cue may therefore also reflect visible task difficulty in the shared game view or verbal communication, not the physiological information alone.
Collaborative XR spaces. Avatar placement in social AR is sensitive to the available physical space and meeting context [6], while digital-proxemics research examines how display and audio affordances relate to personal space, proximity, and attention in virtual collaboration [17]. These findings motivate studying avatar-associated physiological halos in collaborative AR/VR rather than assuming that a fixed desktop HUD transfers unchanged: where a halo is placed relative to an avatar or seat, how proximity and attention change its visibility, and who in a shared space may see it. A future evaluation could compare HUD-based and spatially anchored cues during joint tasks, measuring cue interpretation, support response, workload, and perceived invasiveness while examining proximity, occlusion, and audience-specific sharing permissions. This is a proposed extension: the current desktop-game studies do not establish effectiveness in XR.
Limitations. Recognition accuracy was moderate, and middle states were harder to separate than the endpoints. Study 2 used one wearable ecosystem, scripted pressure events, short missions, and 12 team-level units without an a priori power analysis for team-level comparisons, so the nonsignificant Icon–Numeric difference is not evidence of equivalence. The reported results do not establish that teammates correctly interpreted the physiological cue during individual scripted pressure events. Communication was coded by a single primary coder without a verified reliability sample. Both studies used convenience samples, and consent and data ownership remain open questions.
6 Conclusion
Across two independent samples, a wearable-driven, label-free physiological cue made remote teammates’ HR/HRV states readable above chance and accelerated support relative to No Cue. Compared with explicit numerical disclosure, the Icon Cue showed no statistically significant difference from Numeric Cue in teammate awareness, with lower workload and perceived invasiveness, while Numeric Cue yielded higher coordination usefulness and trust. These findings position ambient physiological cues as a complementary, privacy-sensitive layer for distributed coordination rather than a replacement for precise physiological information.
6.1 Ethics / Data / AI Use
Ethics. Both studies were approved under a single institutional ethics board approval before data collection. Participants gave informed consent and could withdraw or stop physiological sharing at any time; HR and HRV data were used for research and interface visualization only, not diagnosis, and were stored under coded participant identifiers. Data. Anonymized data, stimuli, and analysis scripts will be released upon publication. AI use. Generative AI tools (ChatGPT and Claude Code) were used for language editing and restructuring of author-drafted text. During this revision, ChatGPT also assisted with identifying related literature, and Claude Code assisted with LaTeX formatting and checking reference metadata. These tools were not used to generate participant data or reported statistical results. The authors verified all sources, reviewed the final text, and take responsibility for it.
References
- Karen Barad. 1996. Meeting the Universe Halfway: Realism and Social Constructivism without Contradiction. In Feminism, Science, and the Philosophy of Science, Lynn Hankinson Nelson and Jack Nelson (Eds.). Springer Netherlands, Dordrecht, 161–194. https://doi.org/10.1007/978-94-009-1742-2_9
- Geoffrey C. Bowker and Susan Leigh Star. 2008. Sorting Things Out: Classification and Its Consequences. MIT Press, Cambridge, MA, USA.
- Margaret M. Bradley and Peter J. Lang. 1994. Measuring Emotion: The Self-Assessment Manikin and the Semantic Differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49–59. https://doi.org/10.1016/0005-7916(94)90063-9
- Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
- Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the Utility of Learning about Humans for Human-AI Coordination. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019). Curran Associates, Inc.
- Diego Drago, Joseph O'Hagan, Fahim Kawsar, and Stephen Anthony Brewster. 2025. Exploring Avatar Seat Allocation Strategies in Social AR. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems(CHI EA ’25). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3706599.3719867
- Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist Studies 14, 3 (1988), 575–599. https://doi.org/10.2307/3178066
- Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, Amsterdam, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
- Mariam Hassib, Daniel Buschek, Paweł W. Woźniak, and Florian Alt. 2017. HeartChat: Heart Rate Augmented Mobile Chat to Support Empathy and Awareness. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(CHI ’17). Association for Computing Machinery, New York, NY, USA, 2239–2251. https://doi.org/10.1145/3025453.3025758
- Noura Howell, Laura Devendorf, Rundong (Kevin) Tian, Tomás Vega Galvez, Nan-Wei Gong, Ivan Poupyrev, Eric Paulos, and Kimiko Ryokai. 2016. Biosignals as Social Cues: Ambiguity and Emotional Interpretation in Social Displays of Skin Conductance. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems(DIS ’16). Association for Computing Machinery, New York, NY, USA, 865–870. https://doi.org/10.1145/2901790.2901850
- International Organization for Standardization. 2014. Graphical Symbols – Test Methods – Part 1: Method for Testing Comprehensibility. Standard ISO 9186-1:2014. International Organization for Standardization, Geneva, Switzerland.
- Júlio Castro Lopes, João Vieira, Isaac Van-Deste, and Rui Pedro Lopes. 2023. An Architecture for Capturing and Synchronizing Heart Rate and Body Motion for Stress Inference. In 2023 IEEE 11th International Conference on Serious Games and Applications for Health (SeGAH). IEEE, 1–7. https://doi.org/10.1109/SEGAH57547.2023.10253815
- Sarah Morrison-Smith, Lydia B. Chilton, and Jaime Ruiz. 2021. AmbiTeam: Providing Team Awareness Through Ambient Displays. In Proceedings of Graphics Interface 2021(GI 2021). Canadian Information Processing Society, 20–27. https://doi.org/10.20380/GI2021.04
- Rosalind W. Picard. 1997. Affective Computing. MIT Press, Cambridge, MA, USA.
- Chiew Seng Sean Tan, Kris Luyten, Jan Van den Bergh, Johannes Schöning, and Karin Coninx. 2014. The Role of Physiological Cues during Remote Collaboration. Presence: Teleoperators and Virtual Environments 23, 1 (2014), 90–107. https://doi.org/10.1162/PRES_a_00168
- Julian F. Thayer and Richard D. Lane. 2000. A Model of Neurovisceral Integration in Emotion Regulation and Dysregulation. Journal of Affective Disorders 61, 3 (2000), 201–216. https://doi.org/10.1016/S0165-0327(00)00338-4
- Julie R. Williamson, Joseph O'Hagan, John Alexis Guerra-Gomez, John H. Williamson, Pablo Cesar, and David A. Shamma. 2022. Digital Proxemics: Designing Social and Collaborative Interaction in Virtual Environments. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 423, 12 pages. https://doi.org/10.1145/3491102.3517594
This work is licensed under a Creative Commons Attribution 4.0 International License.
ISS Companion '26, Turin, Italy
© 2026 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-3011-5/2026/11.
DOI: https://doi.org/10.1145/3845796.3859311