PM4Music: A Scriptable Parametric Modeling Interface for Music Visualizer Design Using PM4VR

The intersection of signal processing technology [6, 39, 40] and creative digital expression [4, 42, 47] has led to the development of innovative tools in the realm of music visualization [37]. As a consequence, the fusion of technology and artistic expression has given rise to innovative platforms that bridge the gap between sound and sight [44]. Music visualizers play a pivotal role in this convergence [38], creating a canvas for translating audio frequencies into mesmerizing visual representations [8]. In the end, the convergence of music and visual arts has paved the way for immersive experiences through music visualizers [3], offering a synthesis of auditory and visual stimuli. As an interdisciplinary phenomenon, music visualization intertwines the auditory and visual realms [5], presenting a narrative that reflects the paradigms of artistic expression and technological innovation. This paper seeks to unravel the music visualizer's rich tapestry by examining parametric artistic forms which shapes its visual effects in virtual reality.

Virtual Reality (VR) [45] refers to a computer-generated simulation of a three-dimensional environment that can be interacted with in a seemingly real or physical way by a person, using electronic devices such as headsets, gloves, or motion sensors [43]. The goal of VR is to immerse the user in a simulated world that feels as close to reality as possible, creating a sense of presence and allowing users to explore and interact with the virtual environment [2]. VR headsets are the primary hardware used to experience VR [1]. Advanced computer graphics technologies are crucial for creating realistic and immersive virtual environments [9, 11, 17, 22, 26, 27, 28]. VR has been applied in immersive interactive gaming for entertainments [7, 13, 41], simulations [21, 23] and trainings [46] across various fields, including civil engineering [33, 34], healthcare [35], sports [15, 16], exercise [20, 29, 36], etc. It is also used in education to create immersive virtual environments for enhanced learning experiences [12, 19], as well as in artistic designs [10, 25] for visualizing creations before physical construction or realization.

This paper introduces PM4Music, a scriptable interface for music visualization within the framework of PM4VR [14, 30, 31, 32]. PM4Music allows artists to create parametric visualizations [18, 24] that synchronize with the auditory experience. Serving as a novel solution, PM4Music employs scriptable parametric modeling to facilitate the design of music visualizers in PM4VR's dynamic landscape. Fig. 1 provides a glimpse of PM4Music, highlighting its scriptable parametric modeling interface for music visualizer design. Designers write Java^♭ scripts to generate abstract 3D shapes, seamlessly integrated with the Oculus Quest 2 VR headset for immersive parameter tuning experience using VR controllers in virtual environment.

PM4VR Interface. As proposed by Li et al. [14], PM4VR is a highly effective programming interface integrated into Unity Editor with a specific focus on facilitating parametric modeling in virtual reality. PM4VR contains two crucial C# script files including JavabCompiler.cs, and JavabScriptBehaviour.cs. JavabCompiler.cs establishes a connection between the Unity Editor and Java^♭ compiler. This real-time intercommunication is invoked by a Java^♭ function called getRange("Range Variable Name") and the connection is realized via exchanging Unity^♭ Scripts (*.unityb) between JVM^♭ and Unity^♭.

In assessing the effectiveness of our proposed methodology, we undertook numerical experiments using the scriptable parametric modeling interface of PM4Music. The implementation of these experiments was executed within the Unity 3D framework, specifically the 2019 version. The computational tasks were performed on hardware equipped with an Intel Core i5 CPU, 32GB DDR4 RAM, and an NVIDIA GeForce GTX 1650 4GB GDDR6 Graphics Card. This hardware configuration ensured a robust environment for our evaluations, allowing us to gauge the performance, efficiency, and overall viability of our proposed approach in the realm of music visualizers. The chosen specifications offer a balance between processing power and graphical capabilities, aligning with the demands of the Unity 3D environment and the complexities associated with the scriptable parametric modeling interface of PM4Music.

Figure 3 visually showcases the outcomes of parametric music visualizers created through the innovative capabilities of PM4Music under distinct parametric functions. In this depiction, four different parametric music visualizers are presented in four subfigures, each crafted using a different parametric function which are (f₁(u, v),..., f₄(u, v)). The diverse visualizations highlight the flexibility and versatility inherent in PM4Music, demonstrating its capacity to generate varied and expressive music-driven visual experiences. To provide further insight into the underlying mechanisms, the corresponding Java^♭ script for each of these four parametric functions is detailed in Figure 2. This dual representation offers a comprehensive view of both the visual outcomes and the associated scripting, providing a valuable resource for understanding the correlation between parametric functions and the resulting music visualizers’ 3D geometric models in the context of PM4Music.

The visual representation in Fig. 4 provides a dynamic showcase of the evolving shapes of the parametric music visualizer, whose corresponding parametric equation is detailed in Figure 1, at different time intervals. Comprising 12 subfigures arranged in a 4 by 3 grid, these screenshots capture the visual transformations of the music-driven graphics, with each image representing a distinct moment in time. The snapshots were captured at regular 2-second intervals, covering the temporal span from 2 to 24 seconds. The accompanying musical backdrop for this illustrative example is a piano composition titled "Bach in G Minor" (Arranged by Luo Ni).

User Study. In Fig. 5, we offer a glimpse into the immersive realm of VR, where a user engages with the PM4Music interface while donning an Oculus Quest 2 headset. This figure provides a firsthand look at the experiential aspect of tunning the music visualizer's parameter values within the PM4Music environment, showcasing the seamless integration of VR technology with our proposed parametric modeling interface. To offer a more comprehensive understanding of this immersive user experience, allow readers to witness the user's interaction in real-time, and provide a more in-depth perspective on the music visualizer's VR-enabled parametric modeling approach within the PM4Music framework, we have created a video available at the following link: https://youtu.be/BczA3i2tflA

This paper introduces PM4Music, an innovative scriptable parametric modeling interface for music visualizers design within the PM4VR framework. The interface empowers users to script and generate intricate visual representations that respond dynamically to changes in music parameters. By providing a flexible and scriptable parametric modeling interface integrated seamlessly with PM4VR, PM4Music leverages the power of parametric modeling to dynamically synchronize visual elements with the accompanying music, marking a significant advancement in the realm of music visualization. The paper highlights the architecture, implementation details, and immersive experiences within VR, underscoring PM4Music's potential as a transformative tool for artists and enthusiasts to craft visually stunning and responsive representations of music. As an evolving technology, PM4Music stands at the intersection of art and VR, offering a unique platform for creative expression.