Collective Craft: How Artists Collaborate to Train AI-based Audio Synthesis Model for Music

4.1.1 Epistemic and Aesthetic Mismatch: AI's Reduction of Embodied Musical Knowledge. Participants consistently described a mismatch between the epistemic assumptions embedded in ML models and the forms of musical knowledge mobilized in artistic practice. Sound synthesis systems often privilege abstract, quantifiable parameters, marginalizing embodied, culturally situated forms of musical knowledge. As a result, artists often experience AI not as neutral tools, but as systems that actively reshape creative possibilities. As JCR explains, the vocal synthesis model they employed encodes musical features according to criteria that artists themselves do not consider relevant, resulting in aesthetic biases:

They have implicit modeling of speech features, and anything around voice synthesis almost always involves some kind of F0 estimation. Jaap's music is not about pitch, and I wanted to make sure I didn't bias the model.JCR

This highlights how engineering conventions can constrain artistic intentions from the outset of model design and training. This reduction becomes even more pronounced when dealing with non-Western or traditional musical practices, where expressivity often lies in subtle variations. As reported by WW, working with traditional Chinese music:

Chinese music pitch is not exact. We value raw natural sound. We had to be careful because models assume a very different tuning grammar. If pitch and timbre are too stable, the instrument sounds dead to a professional erhu player. Traditional Chinese music includes many subtle noises, variations, and environment-dependent resonances. These are often one-time experiences that don't translate well into model training. WW

Artists appear to adopt two distinct attitudes toward this epistemic and aesthetic mismatch between optimization-driven learning and artistic intention. The first corresponds to what ACRS describes as a convergent approach: guiding the model towards a result or aesthetic that aligns with the artist's expectations, while negotiating with the technical constraints of the system to seek a compromise. MJ articulates this perspective through her collaboration with two instrumentalists, where models were intended to approximate each performer's style:

When I trained the models on his data, it took much longer to converge than with Matty's data. I created a WaveNet model for each performer, trained on their own data, using Google Cloud. Some generated sounds were still a bit strange, but most sounded like the performer MJ

Artists strive to build models that capture or support their personal musical practice, and they underline the difficulty of translating between epistemic frameworks, particularly when communicating across artistic and technical domains.

By contrast, other artists deliberately move away from this principle of convergence, favoring a divergent approach. Such attitudes can even seek to displace the artist from their habitual sonic identity, using AI to generate forms that exceed or diverge from their established musical vocabulary. ACRS emphasizes this approach by deliberately embracing failure and incorporating artifacts as part of the work itself:

I really distinguish between a convergent use, which is like: AI is a tool, there's this positivist discourse where it reconstructs parameters extremely well, it's absolutely amazing, it has to work that way. And then there's a divergent approach. In fact, the divergent approach even includes artifacts and playful uses, and you integrate those into the evaluation of the object. And in our case it's completely divergent, because in the end we train a model on DAIM, but if it doesn't produce DAIM, we don't careACRS

The training of sound synthesis models is fundamentally based on optimization algorithms designed to drive the model toward convergence on a specific solution. As JA notes, these systems become particularly valuable when approached as “tools for misinterpretation and weird extrapolation”, opening up alternative aesthetic possibilities beyond convergent optimization goals.

4.1.2 Establishing Communication within the Collective to Align Meaning, Interpretation, and Artistic Intent. As shown above, ML models’ representations of sound often clash with artists’ aesthetic practices. When AI enters artistic collectives, technical vocabulary proves inadequate for communicating artistic intent and interpretation of model behavior. As SJZ notes, “explaining technical concepts in accessible terms is hard.” and requires “constantly adapting the language”. Similarly, there is a “barrier in terms of vocabulary” for SN, even considering that “the language itself, verbal language, was not a support for communication.”

We found that artists employ diverse strategies to improve communication within their collectives. In the piece entitled “Prelude”, a duo consisting of a musician and a dancer controls a ML model together through two modes of interaction, a computer and a controller for the musician and movements for the dancer. SN explained that they needed to “communicate through examples rather than verbal explanation”. She developed a visual representation based on 2D Uniform Manifold Approximation and Projection (UMAP), to make data and model behavior intelligible:

We almost developed our own way of communicating. It's not telepathy, but a mode of communication specific to us that gradually emerged. What happened was that I projected my interface with the UMAP, because she really didn't understand the latent space at all, what was happening, what the descriptors were. Having a visual support, where you can see things moving, curves evolving, allowed her to see the correlation between a gesture I make and what happens on the interface.SN

Understanding is not achieved through verbal communication alone, but through interactions between embodied exploration, visual and auditory feedback.

One project developed a shared language in a particularly explicit form through a dedicated notation system. In “Bla Blavatar vs Jaap Blonk”, the artists used a custom real-time voice synthesis instrument (Tungnaa) trained on recordings of Jaap Blonk's poetic performances. Because Blonk's vocal practice was highly codified, JCR explains that they needed to “develop a new notation system”, which was continuously refined throughout the project:

The first notation system I proposed was very abstract. [...] It was describing physiological forms of the vocal tract so it's like open voice, closed voice, upper palate resonance, forward palate resonance, back palate resonance, and then he would had some phonetic cue, saying what to perform, maybe like a “u” and double dot or something like that, and then this long string of weird characters. And he was like: “this is too open. This is too flexible”. So I said, Ok Jaap, Can you make a set of symbols, like no more than 20 symbols, that would capture 95% of what you do, the sounds that you make”JCR

These examples show that artistic collectives actively invent new visual, embodied, and symbolic languages that make collaboration possible, allowing meaning, intent, and aesthetic judgment to circulate between humans in ways that standard technical vocabularies cannot support.

4.1.3 Co-Constructing New Frameworks and Notations to Communicate with AI. These modes of communication not only facilitate collaboration among humans but also enable communication with the ML models themselves. SN used their UMAP visualization to directly shape the model training process through spatial and visual interaction rather than automation and fixed metrics: “I started dissecting each latent space to understand what we needed, what we should keep, what we shouldn't keep. And for that, I used our UMAPs, exploring them manually. That's how I re-trained the model.” The artist does not merely inspect the model but actively constructs an interpretive framework that allows them to decide what parts of the model's internal representations are meaningful, undesirable, or worth preserving.

Similarly, in other collectives, the construction of shared frameworks for communicating with AI is inseparable from cultural mediation. WW highlights how collaboration with HQ was essential to aligning the model's training process with the aesthetic and cultural values of Chinese traditional music:

Hanyu has helped me a lot especially in terms of cultural knowledge and the development of Chinese traditional culture. Her insights into what is unique in Chinese traditional music, including music theory and aesthetics, helped me understand how culture is embedded in sound. So we were very careful in how we trained the model, especially around pitch and expression.WW

In this case, training the model becomes a site where cultural knowledge is actively negotiated and inscribed into the system.

The notation system described by JCR to transcribe the musical style of Jaap Blonk is also employed in the training process, and more specifically in the creation of scores that serve as the foundation for data collection and model refinement: “With this phonetic alphabet, the collaboration loop would be, I would write a score, Jaap would record it, and then we would retrain the model or fine tune the model. [...] This notation might make the model more controllable.” Controllability does not emerge from direct parameter tuning, but from the establishment of a common language across performers, notation, and model. In this sense, notation is less a static representation than an operational language: it structures how the model learns.

Together, these examples show that co-constructing languages, notations, and interpretive frameworks is a central strategy through which collectives make AI models musically meaningful, culturally situated, and responsive to artistic intent.

4.2.1 Training models: an iterative process guided by aesthetic goals. For all collectives, training a model is never a linear process in which data collection is completed first, followed by a single round of training that produces a model immediately fit for use. In practice, participants rarely know in advance what to expect from a model, and because training is undertaken within an artistic context, model quality is not determined by technical benchmarks but by aesthetic and artistic criteria. As VS illustrates: “ Individual training runs are opaque and unpredictable, but by repeatedly training, evaluating, and adjusting parameters or datasets, we could gradually guide the system in a useful direction.” Evaluation criteria emerge through embodied experimentation, making training an iterative feedback loop between data collection, training, and evaluation:

It's better to quickly make some shitty data and train a model than spend months trying to make the perfect dataset and then train. Because the training process and interacting with the result teaches you how to make the dataset. You need the full feedback loop. JA

JA emphasizes that experimentation is more than just refining outcomes; it is a critical step in which the model reveals its potential uses to the practitioner, shaping the evolution of the training process. To guide this process, evaluation is often collective. Members share examples, conduct in-person testing sessions, and discuss results together, as illustrated by MJ: “During training, I periodically pulled down the model, generated audio, listened, and sent examples to Matty and Louis. [...] I visited them around January 2023 with early sketches. They tried them, gave feedback (“this works, this doesn't”), and sometimes sent recordings. I revised based on that.” These collective feedback practices reinforce the idea that training is not an individual technical task, but a shared, conversational, and exploratory practice through which both the model and the artistic intentions are progressively shaped. Stopping this iterative process is itself an artistic decision, one that is difficult to quantify or formalize. In practice, pragmatic constraints such as time limitations imposed by residencies or production contexts often bring experimentation to an end. As SN explains, the interruption of experimentation was primarily dictated by time pressure: “That stopping point was mainly determined by time constraints, unfortunately, because we only had three months”.

The only exception to this iterative logic appears in the DAIM project, where the artistic approach often deliberately consists of using one-shot trained models regardless of the outputs they produce. Here, failure, artifacts, and malfunction are not signs of inadequacy but central components of the artistic evaluation itself as ACRS explains: “We make a lot of sense of listening to the waste, to what is considered not to work”

4.2.2 Sculpting the dataset. Artists engage in what can be described as a meticulous practice of dataset sculpting. Rather than treating the dataset as a fixed input prepared once and for all, collectives describe an ongoing process of manual sorting and refinement, work they consistently considered tedious and time-consuming. As JCR notes, “the limitation is the amount of labor required to clean up the datasets and notate and like go back and fix annotation errors and stuff like that. That takes a long time. and a lot of energy”. Similarly, SJZ describes this process as “trial-and-error” where the artist “spend a lot of time testing different configurations”. Concretely, this can consist in removing peripheral sounds like “ambient sound or other sound sources” (JA), discarding “sounds that were too quiet” (MJ), or pursuing a reduction and clarification of musical information, as articulated by WW: “I refined my dataset and recorded one technique per pitch range, trying to keep each sound as consistent as possible. I wanted to minimize the number of concepts the model had to learn. After retraining the model with this cleaner dataset, I was more satisfied”. Here, dataset sculpting operates as a form of conceptual compression, where artistic decisions about what matters musically are directly translated into the structure of the training data. In other cases, artists reshape the corpus by introducing specific sound qualities, as VS describes: “I suggested reducing long, repetitive utterances and adding more phonetic variation. The process evolved continuously.”

Crucially, this manual and empirical approach is not framed as a flaw to be eliminated, but as a form of creative craftsmanship. This hands-on process of dataset sculpting is clearly illustrated by SN's account:

I kept removing data little by little. It was very tedious and very time-consuming. And at the same time, it was enjoyable, I think, because there's still that craft-like, hands-on aspect to it. I think it was both frustrating and also quite pleasant to keep that manual side of things, to be able to say, “oh right, I matter in this process too.” It's not just the model learning everything on its own

SN's description emphasizes the affective ambivalence of the process, simultaneously “tedious” and “time-consuming”, yet also “enjoyable” and “pleasant”, suggesting that value lies not only in the resulting model but in the sustained interaction with the material itself. Importantly, this perspective underlines how such hands-on labor functions as a way of asserting artistic agency over an otherwise automated pipeline. By insisting that “I matter in this process too”, SN reframes training as a dialogic relationship between the artist and the model, resisting narratives in which learning is entirely delegated to the algorithm. JA's account further clarifies how artistic agency is exercised at the level of dataset design: “when you create the dataset, you establish which patterns you're interested in. Some patterns won't be semantic, but if you wanted to, you could make a flute dataset where you only play the note C. So the agency is treating it as pattern recognition: as an artist, I want to do X, Y, Z, what data patterns bring about that possibility space?”.

Although model training may appear to be a highly automated process with limited opportunities for direct artist intervention, our findings show that dataset sculpting serves as a crucial way for artists to reassert agency within this automation. By focusing attention on the selection, pruning, balancing, and transformation of training data, artists intervene upstream in the learning pipeline, where their actions have decisive influence over the model's behavior. Seen through this lens, dataset sculpting constitutes a key site of appropriation and a means of maintaining authorship, transforming training from a background operation into a performative and expressive component of the creative process.

4.3.1 Building a Practical Understanding of Model Properties. The sound synthesis models used by artists are often complex DL systems whose behaviors and limitations are not immediately transparent. Regardless of whether artists initially have technical expertise in ML, all participants develop a practical understanding of model properties through use. FN describes this learning process as inseparable from collective musical practice, where abstract ML concepts become intelligible:

Because, as I said, it was a new thing for me and I was trying to isolate all this very complex, you know, personal processes. Like, for example, training is a very complex process that one needs to understand. And for me, it was during this feedback or this musical discussion. [...] What I understood specifically was this idea of overfitting and underfitting, but in a very tangible sense.FN

Here, notions such as overfitting and underfitting are no longer treated as technical concepts, but as audible and experiential phenomena that contribute to building a tacit, practical form of ML knowledge. WW reports a similar form of learning grounded in listening and experimentation, where model behavior is understood through its sonic outcomes: “Overtraining can also degrade sound quality. If you train on C3–C5 and play C2, the output becomes noisy.” Training ranges, extrapolation limits, and generalization capacities are learned as material constraints that shape musical possibilities.

Similarly, artists experiment with the models’ ability to produce sounds that are more or less faithful to the training material, as well as with their capacity to generate diverse or unexpected outputs. For example, HS articulates a trade-off between variability and sound reproduction: “There wasn't much variability, which I actually found funnier. The model worked well for reconstruction, but exploring the latent space was actually less interesting for us”. SN describes how this understanding of model's affordances emerged gradually during rehearsals, guiding them to use various specialized models crafted with different datasets:

We had an initial phase where we trained on a large dataset that, in our view, contained a lot of diversity. But we realized by playing with that we couldn't actually control that diversity, it became a kind of large cacophony. In a second phase, we separated three complementary sound environments and trained a specific model for each one, in order to guarantee a quality of expression and a degree of expressive freedom for the performance. That's a constraint that emerged gradually over the course of training the models. SN

This realization reflects an experiential discovery that a single model could not meaningfully capture heterogeneous sound categories while maintaining effective control. Rather than being a design decision made in advance, model specialization emerged as a practical response to the limitations of RAVE models, revealed through use.

WW describes a complementary strategy to regain control over the system's expressivity. Rather than modifying the training corpus or architecture, the artist focused on shaping the interaction layer, “filtering motion signals, stabilizing gesture-to-sound mappings, and introducing different performance modes” to adapt the behavior of the instrument in real time. One such mode “quantized pitch to a scale but still allows glissando between notes”, enabling the performer to move between “tremolo, glissando, or more stable tones” by adjusting parameters and switching modes. As HQ notes, this “rule-based approach relies heavily on knowledge of instrumental technique” illustrating how domain-specific performance knowledge can be embedded into control structures to make neural synthesis systems both expressive and playable.

In sum, artists develop a practical, situated understanding of models’ properties and behaviors. This knowledge emerges through aesthetic judgment and musical engagement, as they navigate competing priorities: synthesis quality, sonic diversity, and model controllability.

4.3.2 Musical practice as a method to Evaluate Model Behavior and Guide the Training Process. For artists, musical practice serves as a method to evaluate ML models using situated, aesthetic, and embodied criteria, which, in turn actively guide the training process. By playing with, listening to, and responding to the system, artists establish what counts as a successful or problematic behavior for them, and translate these judgments into concrete training decisions.

A recurrent configuration is that a performer within the collective explores the system through improvisation, while other collaborators attend to this interaction and interpret it as evaluative feedback. FN describes how feedback from her collaborator's felt experience of playing with the system directly informed retraining choices:

“it was mostly, we would do something, and I would ask, okay, did you feel like you were improvising? Was it following you too much? Was it not following you too much? And then these, I think the first version of Enacteur developed from all these feedback that I got, and 90% of them were from Hugo.’’FN

Here, criteria such as feeling followed, excessive reactivity, or freedom to improvise function as tacit evaluation metrics. In other collectives, this evaluative process is more explicitly collective and relational. In the DAIM project, training is explored through what the artists describe as jamming with the model: three collaborators gathered around a single computer, experimenting live and reacting to each other's responses. Rather than pursuing fixed objectives, musical interaction serves to elicit reactions that function as shared indicators of value:

“In general, we're three people around one computer. It's very jam-like, but in front of each other. And because of that constraint, we tend to experiment live, which makes things even more chaotic [...] But since we're very oriented toward music production, we only have a single screen. So we sit next to each other, and usually, if Axel or Hugo laughs, that means I'm probably heading in the right direction”KR

In this setting, affective cues, laughter, surprise, shared excitement, act as informal but salient evaluative signals. Musical practice becomes a way of collectively calibrating what the system affords and what directions are worth pursuing. Training is less guided by abstract performance metrics than by the model's capacity to generate responses that resonate with the group's shared sensibilities. Evaluation is enacted socially, through musical interaction, and training decisions follow from these moments of collective recognition.

Performance itself can become the primary site of evaluation and guidance. JCR described a workflow in which dataset construction, model training, and performance are folded into a single continuous practice:

“I make a score, we perform it live, we record it live. That way all this work that's going a dataset becomes a performance. Whatever the latest model is that's what we perform with [...] It has also been really interesting in terms of influencing how I compose the pieces, because I need to compose them in a way that they have an arc to them. They have a story that they're telling. So I can't just randomly generate lines of notation. As I really think about how this tells a story to the audience, how it gives us sonic possibilities that make sense in a sort of meta-compositional structure. And also to communicate to the audience what dataset making looks like or sounds like”JCR

Here, performance conditions introduce narrative, temporal, and aesthetic constraints that become evaluative frameworks for both the model and the training process. The presence of an audience foregrounds questions of coherence, expressivity, and meaning, which shape decisions about what data is collected and how the system is refined. Rather than converging toward a finalized or optimized model, training is guided by the ongoing assessment of what the system can meaningfully support in performance. Evaluation and training are inseparable from artistic presentation.

Across these cases, musical practice operates less as a means of mastering the model than as a method for defining, testing, and refining criteria to evaluate it. Improvisation, jamming, and performance serve as embodied evaluation procedures through which artists assess how the system listens, responds, and transforms sounds.