Bend to: Explore the Proprioceptive Interaction Between Plants and Post-humans in an Immersive Experience

a) Experimental set-up: two cameras on the top and front, the light on the right: b) Front view under the UV-B light, c) Top view under the blue light.

To collect empirical data on the response of plants to light, time-lapse cameras were used to document ongoing alterations in unilateral light experiments on 2-week-old soybean seedlings (Figure 4). For the choice of light, 302nm UV-B and 450 nm blue light spectrum have been proven to induce the strongest and fastest phototropic action [11, 13]. All experiments were carried out in a darkened room at a temperature of 22 °C. The 6W blue light (450-460nm) and the 8W UV-B light (302-312m) are placed separately on the right side of the soybean at a distance of 30 cm for 3 hours of irradiation. Two Canon EOS R5 cameras with 50 mm focal length macro lenses are positioned 30 cm in front and on top of the plant, taking an image every 5 seconds to collect data.

The TrackMate plugin of ImageJ was employed for keypoint detection and tracking in time-lapse images to produce positional data changes (Figure 5). Front-view images were employed as the primary data source, with top-view images used for auxiliary validation. A high-quality dataset was obtained by optimizing parameters such as point quality and size thresholds and performing the data cleaning according to predefined data integrity criteria to ensure that the detected points accurately covered the key areas of the plant. The spatial position data of phototropism was acquired by calculating the difference between the x, y, and z-axis coordinates at the start and end of light irradiation. During the modeling phase, a nonlinear Support Vector Regression (SVR) model and time series analysis were applied to predict the phototropic response rate of seedlings. By capturing temporal dependencies, time series features improved the SVR model's accuracy in predicting curvature dynamics under unilateral light stimulation. The obtained rates were classified as fast, medium, and slow into three categories to provide robust support for data visualization in the subsequent project.

Keypoint data of detection and tracking in ImageJ.

One soybean seedling in SHINING 3D software.

Initially, different soybean seedlings were scanned by a 3D scanner, EinScan Pro HD, to get high-quality FBX models. In Unity, through baking those models to “Signed Distance Fields (SDFs)” as the properties of “Visual Effect Graph (VFX Graph),” generated dynamic particle lines in the shape of soybean seedlings (Figure 6). In terms of data mapping, the processed data was used to control the phototropic response rate and the target point coordinates. The process of phototropic bending is fully automated by “C# Scripts” and is based on three different rates provided by the data processing. Meanwhile, a series of “Null Objects” based on the spatial position data was created, providing every seedling with unique relative coordinates for the intended bending point. Afterward, these dynamic soybean seedlings were separately placed in the post-human scene for further interaction.

a) The user wears a VR headset for the experience. b) The instructional interface shows texts and semi-transparent circular shapes.

The VFX Graph necessitates rendering in the High Definition Render Pipeline (HDRP), which is not supported by stand-alone VR headsets such as the Oculus Quest 3S. Consequently, this project operates on a PC with Meta Quest Link via Steam VR to facilitate interaction. It also offers a feasible solution for executing generative art produced by VFX Graph with stand-alone VR headsets in a VR environment.

Bend to offers an immersive and interactive experience in the 3D digital space that can sense the future light irradiation environment composed of phototropic data together with plants. The proprioceptive interaction consists of two phases: “connection” and “blending.” Upon entering the plant's world with the VR headset, instructional texts will be presented to facilitate the experience (Figure 7). Two semi-transparent circular shapes are outlined to signify each hand, assisting users in recognizing their bodily (hand) position. For the benefit of participant immersion, circular figures will not consistently be displayed.

The plant has color alterations and the floating circles effect when users touch it virtually.

Through the VR equipment, users entering the soybean plant world will initially observe different seedlings bending their bodies toward the light at different angles. When users grope around with their hands to search, they will focus on their bodily movement while traveling through different shapes of plants. Once they make a virtual touch with one particular plant, it will provide users with visual feedback in the form of color alterations and the floating circles effect (Figure 8). Through the response of plants, users are “connected” to plants in the virtual world and subconsciously perceive their bodily position in the physical world alongside the position of the plant in the virtual environment.

In the second interactive effect, “blending,” smoke permeates around the edges of the plant, surrounded by spiraling waves of light.

During the second phase, “blending,” users will have the opportunity to observe the phototropic response of plants. When individuals’ hand motion tracking aligns with the positional changes in phototropic bending derived from the experimental data, the “connection” will be reinforced to “blending” as splendor: smoke permeates around the edges of the plant, surrounded by spiraling waves of light (Figure 9). That kind of embodied interaction enables participants to fully sense how plants respond to light radiation and their spatial sensory abilities. Throughout the whole experience, users can adjust their viewpoints to access the insides of plants’ bodies.

The equipment required for the exhibition of this project involves an Oculus Quest 3S, a PC laptop with Windows 10 and above, and a projector or a screen connected to the computer. The recommended installation area should be a square measuring 8 meters by 10 meters to enhance the audience's experience while navigating the virtual scene. This project is flexible regarding the exhibition space and could fit a smaller area if necessary. A quiet and relatively dark environment could provide optimal immersion for audiences.

The exhibition area should ideally have a projection wall or screen to display the real-time moving images captured by a stationary camera in Unity, enabling audiences to observe the other users’ experience from a third-person perspective. The VR headset should preferably connect to the laptop wirelessly via Air Link. In case of poor network conditions, a sufficiently long Type-C to USB 3.2 cable may be used for connecting.

In GatherTown, this work can be presented as a streamed VR scene or a documentation video. More information can be found at: https://yutingxue.com/Bend-to

A user interacting with the VR device is projected on the wall.