Challenges of Music Score Writing and the Potentials of Interactive Surfaces

Bennett [3] conducted one of the first qualitative studies about the musical creation process, interviewing eight professional composers. Each composer has their own process, but high-level activities can be identified across composers. The discovery of a “germinal idea” often comes first. And then the drafting of this idea, elaborating upon and refining it – what is identified as the writing and rewriting phases in another study by Roels [35]. Rosselli del Turco and Dalsgaard [39] provide a good overview of additional studies that aimed at better understanding composers’ creative process [12, 44], and contribute their own study about how music artists capture and manage their ideas. They draw a parallel with personal information management tools, and discuss recommendations for the design of tools aimed at supporting idea management in music.

Key takeaways from the above studies of particular interest here can be summarized as follows:

• even if the creative process can be characterized in terms of overall strategy [12] and high-level activities [3, 35, 44], it varies significantly from one composer to another and across music genres [12];
  
• the process is highly iterative [10], composers writing and rewriting [4, 35], spending much time elaborating and refining their draft [3].¹
  

Studies in the HCI community about composers also emphasize the iterative nature of the music writing process [15, 25, 43]. These studies have had different goals. Following early observations about contemporary composers’ use of paper and computer by Letondal & Mackay [25], Tsandilas et al. developed Musink [43]. Musink is a computer-based composition environment that integrates sketching, gesturing and end-user programming in the same workflow thanks to interactive paper [21]. Garcia and colleagues further explored this line of research, designing creativity support tools that combine paper and computer programming to support contemporary composers’ creative process [13, 14] as well as to observe that process [15]. These works differ from our work in two ways. First, while interactive paper is very intriguing because it faithfully replicates the experience users have with physical paper, it also inherits drawbacks of physical paper. In particular, as opposed to marks on interactive surfaces, marks on interactive paper cannot be modified, moved or deleted once written. Second, these projects focused on a very specific community: contemporary composers. While these composers may use the staff notation to capture some ideas, they primarily rely on music programming tools such as Max [33] and OpenMusic [7], as well as on notations that they design themselves.

Closer to our research objectives, Leroy et al. [24] were among the first to develop an interactive system using an early form of optical music recognition [8] to let composers use handwriting to input music symbols on staves. From the very beginning, Leroy et al. explicitly state that this is “a system for composers, not for engravers,” observing that “composers are skilled in handwritten notation and editing and need to sketch out musical ideas rapidly while retaining the ability to make alterations at both local and global levels” [24]. Presto [1, 30] is another early pen-operated system for music score editing. The system was designed to facilitate the input of music symbols on staves, this time using a set of simple, predefined gestures rather than handwriting recognition. Several other manipulations were possible, such as moving elements, transposing or adding ornaments, but accessible via menus only. Current interactive surfaces have capabilities significantly more advanced compared to these early systems. In this paper, we advocate for a comprehensive reevaluation of the entire interaction model, broadening the scope beyond rudimentary music symbol input.

Coughlan and Johnson [11] discuss the tension between the desire for flexibility of composers and the constraints imposed by software tools that stems from interaction design choices made by the developers of those tools, and from the need for these tools to be able to interpret user input. While they made this observation about composition tools that do not use staff notation as the primary means to represent music, this tension clearly exists in music score editing software as well. Many of the programs used by composers (Finale, MuseScore, Sibelius) impose significant constraints on how composers can input and modify the notation, impeding the creative process as we discuss in detail in Section 5.

Several music composition programs designed for the desktop can operate on interactive surfaces that support digital pen input, including Sibelius, Dorico, Flat, and Notion. A few programs have also been designed specifically to run on interactive surfaces, such as StaffPad [41] and Symphony Pro [32]. But both categories essentially rely on the traditional WIMP model of desktop programs, making very few adjustments to it. Pen and fingers are seen as generic pointing devices that can be used interchangeably, and composition programs treat them as mere alternatives to mouse or trackpad. They do not fully leverage their specific affordances and expressive power.

In this context, the pen is little more than a mouse with a single button, making numerous mode switches necessary. It still has interesting properties though, most notably when inking on a canvas as it better affords handwritten input. The few music composition programs designed specifically for interactive surfaces thus support handwritten input of common music symbols. This capability is highly valuable as it makes the experience closer to writing music on paper. But multiple constraints still impede the creative process. Two constraints relate to the ink recognition process. First, symbols need to be inked in a predefined order to ensure that the recognition engine will be able to interpret them correctly, imposing a particular way of writing on composers. Second, recognition is implemented as a greedy process, beautifying handwritten symbols as soon as a measure is complete and enforcing syntactic well-formedness rules on beautified symbols systematically. For instance, a measure must be filled before proceeding to the next one. The user experience is thus closer to writing music on paper than what desktop programs afford, but the flexibility associated with writing music freely on paper is lost. Those constraints limit composers’ freedom to experiment and jot ideas down [29]. Composers need to follow certain rules when inking the notation, and once input, the notation remains difficult to edit.

Besides music composition, prior work has investigated the use of pen input in a variety of application areas, identifying interesting properties of this modality when combined with touch to manipulate dynamic graphical representations. First, beyond the ability to write and draw freely with an appropriately-shaped input device, pen and touch provide capabilities to make annotations that are well integrated with the primary content in the document [38, 42, 49]. This typically results in an easier externalization of thoughts [26, 28, 34, 40], which is key to many creative processes. The ability to point precisely and delineate arbitrary regions has proven effective when interacting with canvases hosting diverse types of marks and glyphs [36, 47]. Such capabilities seem particularly interesting to enable elaborate selections of music symbols that can be subsequently modified by direct manipulation.

Interactive surfaces also enable contextual gesture-based command invocation [2]. Those commands are usually triggered immediately, but their interpretation can also be delayed [43] by leaving ink traces to be activated later [37], thus avoiding some types of premature commitment [20]. Finally, implementing a division-of-labor strategy between pen and touch can offer a higher level of expressive power compared to interactive systems featuring a single type of pointer [22, 27, 31], increasing the number of actions that can be performed by direct manipulation.

These specific characteristics of interactive surfaces suggest that they have much potential to improve support for music composers’ creativity. But to inform the design of music notation software on those devices, it is first necessary to understand the writing and rewriting phases of their creative process, and more specifically how they use current music writing software, what challenges they face, and how they address them.

In this part, we highlight the importance of details to our interviewees, and the rich interplay of high-level and concrete low-level concerns during the composition process, which echo the different levels of musical abstraction in compositional activities observed by Roel [35]. As AO puts it, “music is an art of details”, with a constant interplay between abstract and concrete levels where music comes to life, emerges. Echoing Nash's observation about composers’ need to “access every detail of a piece, but also be able to get a sense of the ’big picture’,” [29] the important aspect to grasp is thus that there are generally micro and macro levels. Those need to be strongly linked together but have different properties, with much variation between composers. The shape this interplay takes varies depending on the composer. For GF, there is “a bit of dialectic between the macro and the micro,” where “the macro level would be the vision side, the micro level would be the purely auditory side.” For this composer, an abstract level view is required to clarify the state of his composition: “sometimes you can't see, like I said, you've got your head in the sand [...] that's the macro side, that's it, the vision side.” But this is in contrast with MS, for whom reading at the concrete level of music notes helps making ideas clearer, since “there are times when you're going to have to go through the score to see things more clearly.”

For some composers, the micro and macro levels are co-constructed by working on short musical segments. For instance, GF explains how the primary notation level (e.g., rhythm) can also provide a macro level idea, as in “Beethoven symphony number seven, the andante, there's precisely this macro rhythmic side that's <<taa ta ta taa ta>> and you can step back and see it higher and higher, it's always there [...] when you look at it from a macro perspective.” The emergence of a macro level can also come from an amplification of micro elements: “It's a little world in itself that just needs to be enlarged, explored, amplified in as many ways as you want [...] Afterwards, if need be, like a painter, I like it to be stretched out a bit with water.” (GF). The emergence of form is compared by AO to the progressive sculpting of, e.g., attack, resonance, orchestration...; and by LA to polishing thanks to stronger ideas: “once you start to give a form [...] you polish, you polish, you polish, because you have found more expressive ideas, better ideas, stronger ideas, ideas that correspond better to what you wanted to do.”

Details can be immediately-perceived data, pure matter (GF: “ideas for themes come to us, and for us it's that and nothing else. It's obvious, it's pure matter, so to say”) but also constraints, and tedious work in progress (“GF: These places, we say to ourselves, are where there's the most craftsmanship, we're bound to say that it's not as good as art with a capital A and in fact these places are sometimes the ones we prefer too”). What falls in the details category is also what is less essential, as in the case of JP who creates versions of a song with different levels of difficulty. To build an easier version: “the idea is to make a slightly minimalist version of it, where we keep the essence of the song, the essential things and where you remove details or change specific aspects, for instance by keeping only a part of an arpeggio that is emblematic for the piece” (Ständchen from Schubert).

Time, and the associated aspects of linearity and change, play an important role. As AO also says indeed, “music is an art of time,” which raises questions about the experience and sensation of time, but also about its representation.

For the composers we interviewed, the experience and sensation of time can consist either in putting oneself in the place of a listener “there's bound to be listening, a human ear, a human temporality” says MS, or in trying to live through a more linear composition process an experience equivalent to improvisation. In general, the composition process is not linear but rather highly iterative [10, 35], as several of the interviewees explained: “It's never too linear” says JP, “I think that's a bit to be avoided”. So, as highlighted by PG, “it's not written as you go along, starting on the first note and ending on the last”. However, AO explains the need “to get back into the feeling of time and to get back into the improvised side of things [...] because composition isn't improvisation [...] In fact, right now I'm forcing myself to go back to something more linear, so as not to go too far into abstraction either.” GF, for his part, compares composing to a journey: “I see [composition] as a process, as a journey [...] I really like initiatory things, I really like travelling and I think that each composition is a little journey and I think spontaneously I compose rather in a linear way.” This comes with “happy accidents”, as DR puts it: “I like a bit of accidents, surprises and all that,” and a relative reluctance to accept a pre-established structure: “I like to give myself a certain amount of freedom when it comes to travelling.”

The sensation of time also encompasses the perception of change, which is why music often has to provide a sense of novelty, but within a familiar context. For this reason, repetition is important if the listener is to get familiar with the themes. To convey the perception of changes, the themes will be more or less transformed, as noted by Blackwell et al. [4] as well. As GF explains, there is a need to “transpose or change melodic ideas, rhythmic ideas, or patterns.” It is important to think carefully about the number of repetitions, and about the type of those repetitions: “I don't make music that's very repetitive in the sense of the motif, it's not the motifs that are repeated, it's more the sequences, the harmonic ranges, that sort of thing” explains GC.

Regarding representation, our interviewees express a need to “get an idea of what it would look like, for example in terms of time proportions [...] to work out the temporal relationships of different episodes in a composition” (PG). For DR, “finding your landmarks in time is easier with paper.” Other needs encompass landmarks in time, that can be provided either with bars, rhythms, pages (“a page is like a rhythm” as DR puts it) or musical events, and time durations that can be also provided with bars or with seconds. Knowing that “such and such a bar lasts six seconds” is important to GC, and for composers making movie music.

The composers raised concerns about assembly, and how they work towards the shaping of a specific individual. They use all sorts of metaphors to describe how they see and manage the assembly of musical components to make a piece of music, from “gap-fill text” (GF) to “jigsaw puzzle” (MS) or “sorter” (DR). Their concerns relate to being able to put several ideas into dialogue, to mix them: “because you're obsessed with the same things, it comes from you and it's material that will end up fitting together” (GF) or superpose them: “And here, I'm really working on the melodic motif that I'm going to associate with my rhythm, for example. So after a while, the two things will probably overlap. I'll superimpose them on the score, for example” (LA). They also need to design transitions, because “transitions are meant to unify the piece”, as expressed by LA: “It's like making a robot, you build the leg and the arm separately. And then you have to find a way for the leg and the arm to work on the same entity.” Composers indeed build on a more or less clear vision of the specific piece they are working on until they reach a state where something new, something that “works”, comes to life, which they may compare to birth: “But what often interests me is finding, let's say eight, twelve, sixteen bars, of an impactful melody that works with the harmony and that it all comes together as one. That's kind of my ideal, and we'll say that it's a birth, a raw material that already has a lot” (GF). They are constantly on the search for something that will resemble the musical intention that leads them to the specific individual they are interested in: “you have to find something that resembles the idea” (LA), “build it as good as you wanted it to be” (CF), and “make it real” (GF). As we have seen above, a piece is made from a converging process involving low- and high-level elements that feed off each other: high-level, abstract elements (ideas or structure) are the ingredients of an architecture in which lower-level elements (notes, dynamics, rhythm, articulation) can take shape to form this musical individual. This process is well summarised by AO: “In fact, there's matter and form. So it was Aristotle who distinguished between matter and form, if I'm not mistaken, and form is what gives it order. It's what makes it not just a collage of bits of music that we've put together without any vision behind it. Form is what makes it come alive, that it's not just matter, it actually has a shape”.

Transcribing music requires composers to divide their attention between the source – whatever form it takes: original printed score, audio recording, live performance of an idea using an instrument – and the target – typically a score. It is a crucial activity across composers, who rely on multiple tools depending on the type of project and step in the process. These include their mind, paper, an instrument, an audio recorder, a computer. Ultimately, they need to produce a legible score (or set of scores) using the standard staff notation so that it can be shared with and played by musicians. In this section, we discuss the different kinds of transcriptions that may take place during the composition process.

5.1.1 From Immediacy to Computer. Transcribing musical ideas coming from the mind, whether writing them or playing them, is akin to capturing fleeting thoughts. As note-taking, the transcription process needs to be as smooth as possible so as not to impede the flow of ideas [37]. For instance, CF, having established a basis of raw ideas in her mind and on paper, concretizes them by entering them on her computer: “I use [the computer] more like a typewriter.”

However, inspiration can occur in situations where neither paper nor computer are within reach. As DR explained: “Ideas often come like that, when you are not at your desk [...] you take a step back and it comes.” Seeking a linear flow (see Section 4.2), many composers find their ideas while playing their instrument. They do not want to interrupt their flow to write their musical ideas down. They might record themselves using an audio recorder, adding yet another source of information that will later have to be transcribed since audio input is rarely integrated in music notation software. Because they are created independently, these recordings can contain discrepancies with the rest of the music piece when several instruments are intermingled (high level of hidden dependencies), resulting in assembly challenges (see Section 4.3).

Adding small bits of information to the music notation differs from transcription in that it does not require composers to divide their attention. They can remain primarily focused on the score. Moreover, incrementation not only involves adding content – such as notes, symbols or text – but modifying or erasing content as well. In this section, we focus on the context in which composers perform this activity.

5.2.1 The Sacrifice of Flexibility for Aesthetics. A score made with music notation software is both beautiful and well-structured. Resembling an engraved score throughout the work process (providing high visibility and progressive evaluation), it can help composers with some tasks. But it also impedes other tasks. For instance, AO writes a lot of annotations for himself on the score and needs to get rid of them when sharing the final product. He has trouble identifying these annotations because they look too similar to the primary notation (poor support for secondary notation). Another major problem of working directly on the final product and not on a personal draft is the negative impact on creativity, because the rigid framework that comes with this appealing notation imposes rules on composers that sometimes prefer more flexibility (low abstraction management). As GC said: “It's configured in every way, so to be able to unconfigure something is not easy, you have to really dig into the depths of the software, and the user is not at all creative with the tool. He can't create his own tool, he can't develop, except by being super strong. But otherwise it's true that we endure a lot and in fact we can do things but it's always tricky, tinkering.” MS wished for an interface closer to his mental model, that would not hinder his workflow: “I would just like to write it in a more graphic way” (low closeness of mapping).

It is indeed difficult to add notations because composers need to deal with issues of form and position, whether it be within measures (e.g., elements that shift when you add new ones) or around them (e.g., additional lines, ornaments) (high knock-on viscosity). But the visual issues are symptomatic of deeper, more logical ones. For instance, CF commented about being unable to step outside of the rhythmic framework: “The starting point is the measure. But when we don't have measures [in our music], that's a lot more complicated because we are still obliged to pretend we use measures, to delete them afterwards, to ensure that they don't appear anymore and that the spacing is still consistent” (Figure 2-c). LA shares this concern: “In these software, the rhythm is hyper framed. If you make music in 4/4, everything has to be in 4/4 perfectly. It can't overflow. You are forced to solve rhythm problems where sometimes you just want to put your idea. And sometimes it goes beyond the measure.” Such problems, that do not exist on paper, force composers to look for answers in the software's documentation or on the Internet, leading them to think twice before adding a complex notation (low provisionality) or to postpone their action, as GF: “There are places that we deliberately leave under construction, we tell ourselves that we will come back to them later” (high premature commitment). Because phrasing a practical problem to get help from online forums is troublesome, some composers give up and avoid using music notation software to their full potential. On top of that, elements such as nuances or fingerings are also rather added towards the end, in order to avoid having to repeat this task in case of more fundamental changes to, e.g., the rhythm or melody.

The reorganization and restructuring of existing notation focuses on its layout and implies that no new content is added to the score. Composers often have to modify the layout of individual elements or entire measures to improve the readability of the score. But multiple dependencies, between elements and measures and between elements themselves, make this difficult. We detail the three levels at which we observed such dependencies.

It is common practice for composers to play around with new ideas on their instrument and only put them in writing after they are satisfied. However, going through writing for exploratory design without being sure of what will result allows to see and understand the music mechanisms, favoring progressive evaluation. As illustrated by MS: “When there are several voices, it allows you to see how they can fit together so that it can work [...] Seeing the notes in writing makes it clearer in the head. Otherwise it's not intellectualized, and it stays a bit vague.” We discuss several strategies to support exploratory design, each adapted to its situation.

Blackwell et al. [4] identified the search for occurrences of a theme or motif as one of the generic activities performed by composers. More specifically, our observations reveal that composers search for information within the score for two main reasons: when looking for an element – typically to edit it – and when reviewing their work, checking for possible mistakes, making sure everything is “as it should be” – shaping a specific individual (see Section 4.3). We identify search strategies available to the composers and discuss the importance of the chosen perspective.

Our analyses revealed that, in their building of a rich experience happening within time (see Section 4.2), composers dedicate much attention to repetitions in their work, inserting recurring patterns but customizing some or all occurrences of those patterns so that repetitions become variations rather than mere duplicates. Composers work with various aspects of patterns: rhythm, pitch, fingering, nuances, ornaments. They explained heavily relying on copy-paste to facilitate such pattern-centric work but, because copy-paste is monolithic, spending a lot of time to add, remove and adjust elements within the pasted fragment, as we detail in our analysis of their incrementation activities. Furthermore, our examination of their search activities revealed that music notation software lacks effective means to identify musical segments based on specific musical or notation properties.

The precision and flexibility of the digital pen could empower composers when working on patterns. The pen can precisely point at and delineate graphical elements. Current music software only allow monolithic selections, but with this type of precise selection input device composers could tell the system what specific properties they are interested in. This could for instance be through direct selection by, e.g., circling them or making a freeform lasso selection on a specific element in order to restrict the selection scope to only elements of the same type. For instance, they could initiate a selection on a beam (resp. a note head) to copy only the rhythm (resp. the pitch) within the delineated fragment. Or they could select and copy only the fingerings to be repeated throughout the score – an operation that was described as particularly tedious in our interviews. Beyond the direct selection of elements, the pen could leverage users’ ability to write and draw elements by hand beyond input of the primary notation to search patterns in a score and thus facilitate checks and edits across repetitions. Interviewed composers reported the lack of support to search patterns according to diversified musical properties. A pen-operated search box could address this limitation. Composers would be able to express the pattern they seek by drawing note heads, beams, stems, or combinations thereof, specifying a query across their score to identify fragments that match their desired pattern, as illustrated in mock-up (a) in Figure 4. Based on what gets highlighted in the score, they could draw or erase some ink marks in the search box to further constrain or relax their query.

Our interviews revealed a potential mismatch that can occur when the temporal flow for writing music in software is not aligned with the natural flow of composers’ creative thoughts. This creative flow is different depending on the composer's thought level: at a low level of details (see Section 4.1), it is rather temporal and auditory, while at a high level, it is more about spatial organization, considering parts, structure, and relationships, which are predominantly visual in nature. Our analysis of work process revealed that composers often pause their software-based work to turn to alternative mediums such as paper to capture fleeting ideas in the form of notes and sketches, or an audio recorder to capture musical sequences played directly on their instruments. Additionally, they are often constrained by the software's rigidity in terms of form and layout, when they would rather have the flexibility to leave certain sections in draft form for later refinement. In essence, composers’ thoughts cannot always be expressed in the primary notation supported by the software, leading to the usability issues detailed across the Transcription, Incrementation, and Modification activities in Section 5.

Current music notation software expose scores as one homogeneous sequence of staves that cannot be broken and mixed with other content. To address the above issue, the score editing environment could be made more flexible, breaking the homogeneity of the score to support the insertion of different types of contents and media both within and between staves. These elements would not persist on the score. They would exist only temporarily, until they get transformed into actual music notation. Their purpose would rather be to enable composers to insert ideas in-context, regardless of the way they were captured. Starting with a relatively simple case, recent work on supporting in-context annotations on pen-based devices can inspire interaction techniques for music notation software. For instance, SpaceInk [38] enables users to insert handwritten notes between words, lines, or paragraphs in a structured text document by locally reflowing its contents. RichReview [49] also explores the concept of in-context annotations with a rich variety of media, including audio annotations. Such systems preserve the spatial structure and linear flow of documents without being overly constrained by the original document's primary notation. Applying a similar approach to music notation software, composers could for instance insert empty space between existing measures on a staff to jot down a new idea (3rd grand staff in Figure 4-b). They could also insert ideas captured as audio recordings, represented using their audio spectrum (Figure 4-b across the 4th and 5th great staves).² In both cases, the new idea is placed in-context effortlessly, as it does not have to be converted to the primary notation immediately. This can be postponed to a later time, causing less interruptions to the composer's train of thoughts.

Whichever notation we consider, offering flexibility between interpreted input (digitally-enhanced musical symbols) and non-interpreted input (handwritten symbols) seems particularly important. Non-interpreted input lets composers capture fleeting thoughts without disrupting the creative process. Composers may choose to leave a region in the form of annotations or as elements from the primary notation but with relaxed constraints (e.g., without mesure bars) and refine it later (3rd grand staff in Figure 4-b), eventually asking the system to interpret when it is stable enough. This is opposite to what music notation software such as StaffPad does, as it recognizes hand input notation greedily – see Section 2.3. It seems particularly important that composers keep control over when a passage gets interpreted to avoid the system taking premature actions such as adjusting the layout while they are still in a transitional phase of modifications (e.g., when filtering their score for a specific instrument). Pen-based systems such as MusInk [43], WritLarge [48] and ActiveInk [37], which give users much freedom regarding when to interpret pen input, can inspire the design of such interactions.

Linearity is a fundamental aspect of a musical piece. Composers assess their scores with respect to temporal evolution and temporal constraints. However, the creative process itself is often non-linear. It can even extend beyond a single composition. In particular, in their complex and iterative assembly process (see Section 4.3), composers frequently revisit specific sections within their scores to explore alternative versions. They also sometimes draw inspiration from their own previous works, or from the works of others. Unfortunately, music notation software poorly support tasks that involve sections scattered within a score or across scores. Additionally, when revising a section at a detailed level (see Section 4.1), composers often copy the section and paste it right after the original to work on that copy (potentially disrupting the layout). Or they copy it towards the end of the score, potentially losing valuable context. Some composers even prefer creating several independent versions of their score. This is due to the rigidity of the score's linear structure. A well-designed pen-based system could allow temporarily breaking this rigidity to facilitate the activities that this rigidity adversely impacts, as identified in our analysis of the work process: Transcription, Incrementation, Exploratory Design.

One particular advantage of pen-based systems is their ability to replicate the experience of working on a blank sheet of (unstructured) paper. In digital systems, this concept can take the form of an infinite canvas where users can insert various elements and arrange them freely [36, 37, 47]. Designing such a canvas mode would be a valuable addition to music composition software that could support much more operations than the Idea Clipboard depicted in Figure 3-b, which is essentially a list of archived passages that can be pasted on a score. With a more flexible canvas, composers could organize and “retrieve [collected ideas] based on diversified cues” [39]. Figure 4-c illustrates how such a canvas could be used as a storage space as well as a sandbox to experiment with fragments safely. Those fragments could be arranged freely to support spatial grouping and indexing, and to support tasks that involve their comparison side-by-side. This flexibility would for instance help address the challenges composers encounter when transcribing content from one score to another. It would also facilitate exploring different versions of the same passage. With a canvas mode, composers could work on different versions concurrently, add annotations as needed, and then preview and transfer some of them back to the original score, effectively committing their changes. By combining an unstructured and infinite canvas with annotation capabilities, composers could keep a record of their creative process, that would thus favor reflection when needed. Transitions between these views can be made seamless in pen+touch input systems, thanks to, e.g., command marks [2] or menus designed for quick access to frequently used commands [31].

The proposals presented above give a glimpse of how the affordances of interactive surfaces could help address composers’ contradicting needs for structure and flexibility. They were chosen as representative examples but they by no means represent a complete solution. Much remains to be done and developing means to better support the creative process in music notation software will require careful interaction design. As evidenced by the literature on pen + touch interaction, multiple challenges need to be overcome. New and existing features must form a coherent whole. Essential features need to be exposed primarily as direct manipulations seamlessly integrated in terms of input. In addition, the legacy of designs implemented by existing software, that composers have invested significant time to master, cannot be ignored.

An additional challenge lies in the integration of interactive surfaces into composers’ work environment. Rather than an immediate and full replacement for a desktop workstation setup, we envision interactive surfaces serving in two situations: 1) as the device of choice when away from the main workstation (mobile context of use); and 2) as a complementary tool alongside more traditional mediums such as paper and desktop computers [4]. An effective interaction design will likely position them as a preferred option over paper in certain scenarios or over computers in others. We even anticipate that, for some composers, interactive surfaces may replace the need for either paper or computers entirely. Ultimately, interactive surfaces have the potential to enable a more radical change in composers’ work environment where both paper and computers would be replaced by a single, comprehensive, and efficient medium.