EuterPen: Unleashing Creative Expression in Music Score Writing

Pen input has been studied in the HCI literature both as a generic modality and as a device adapted to specific tasks, demonstrating interesting properties on its own and when combined with touch input. Several of these properties are of particular interest in the context of music score writing.

• Digital pen input is both versatile and precise: a pen can be used to select small graphical elements, to draw arbitrary shapes, and to write text. It is an excellent tool to make annotations (writing, sketching) in dense environments [41, 46, 53], to make freeform selections of sets of small items (there is much less occlusion than with touch input [37, 40]); and to perform direct manipulations in 2D canvases hosting rich content [39, 50]. In the context of a music notation program, such capabilities can enable elaborate selections and direct manipulations of the primary notation's symbols, and the input of secondary notations such as annotations [12] – all with a single input device.
  
• Pen input, combined with touch, can have high expressive power: compared to interactive systems that only feature a single type of pointer, interactive surfaces that discriminate between pen and finger touch make it possible to implement a division-of-labor strategy [22] that maps a broader set of direct manipulations to individual modalities, sometimes using one to parameterize actions performed simultaneously with the other [11, 27, 34]. Music symbols can be modified in a variety of ways, that typically require selecting different tools from a palette or invoking commands in a menu. Enabling a broader set of direct manipulations this way can streamline the editing process.
  
• Digital pens are well-suited to handwritten input: such input is not limited to raw ink but can have structure and can be made editable [43]. Beyond notes and sketches, a digital pen has the potential to let composers input music symbols by handwriting as easily as on paper, provided it is properly articulated with the music symbol recognizer that will eventually parse and beautify the handwriting.
  
• Pen input can leave actionable ink traces in the workspace: those traces can be temporary or permanent. They can be used to invoke commands [2], that can be triggered immediately or interpreted later [40, 48] – which can help avoid some cases of premature commitment [19]. Those traces can also be used to generate new content [20, 43, 46, 50], sometimes coupled with an ink recognizer. This can improve closeness of mapping [19] compared to other input methods such as, e.g., keyboard shortcuts or tool palettes, and has the potential to make music symbol input much more effective.
  
• Writing with a pen favors the externalization of thoughts: capture fleeting ideas, take notes [26, 37]. This has cognitive benefits such as lowering working memory load and facilitating idea reformulation. It is particularly important for sensemaking and data exploration tasks [28, 40, 42], but it is also key to many creative processes.
  

Two main strategies have been explored for music notation input with a pen: recognition of handwritten strokes; or recognition of simple predefined gestures from a controlled vocabulary. Leroy et al.’s system [24] was based on an early form of optical music recognition [10] that would let composers input music symbols on staves by drawing them. With Presto [1, 32], composers would input symbols using more abstract, predefined gestures and could then perform some manipulations on them using command menus: transposing, moving elements, adding ornaments. Those first systems paved the way for future ones but were limited by the capabilities of early hardware. Current hardware has significantly more advanced capabilities, providing an opportunity to effectively use pen input and to combine it with touch to broaden music score writing beyond rudimentary symbol input, and to streamline the iterative editing process of the notation.

More recently, another line of research [16, 17, 48] has investigated a particular type of ink pen (Anoto digital pen) equipped with a digital camera to record what gets written on special sheets of so-called interactive paper [21]. Marks drawn on interactive paper are actual ink. They cannot be modified, moved or deleted, making iterations on the notation itself impossible. Targeting contemporary composers, the aim of this research was rather to enable the creation of music pieces by combining programming tools  [8, 36], custom notations and score fragments using gesture-based interaction.

On the commercial side, we find music notation programs originally written for the desktop that can operate on interactive surfaces, such as Sibelius, Dorico, or Flat. These are mostly straightforward adaptations of their desktop counterparts: composers write music symbols by first selecting the proper symbol type in a palette (virtual keypad) and then clicking on staves to instantiate this symbol. They make very few adjustments to the desktop version's WIMP model [12], and treat pen and touch interchangeably for the most part. They rather promote similarity between the desktop and mobile versions of a program. They thus miss an opportunity to leverage the affordances and expressive power of pen and touch combined, and keep relying on menus, tool palettes and mode switches to give composers access to a wealth of features.

Only a few music notation programs are designed specifically for interactive surfaces such as the Apple iPad or the Microsoft Surface Pro. These include StaffPad [44], Symphony Pro [35] and Notion Mobile [33]. These truly leverage the pen, effectively supporting handwritten input of many music symbols and making the music writing experience closer to what it would be on paper. But these programs still impose significant constraints on how composers may input the notation, constraints that are related to the underlying recognition process. In order for the recognition engine to properly parse their strokes, composers must pay more attention to how they draw symbols than they typically would when writing freely on paper. This concerns not only the shape of head, stem and beams, but the order in which they are input as well. A constraint that perhaps more deeply impacts composers’ creative process is related to the temporal articulation between symbol drawing by the user and recognition by the engine. Recognition is typically implemented as a greedy process: as soon as the parent measure is complete, the system automatically beautifies inked symbols and checks that the measure is well-formed: a measure must be filled correctly before proceeding to the next. And once it has been beautified, the notation is difficult to edit: as in desktop music notation programs, syntactic and structural constraints must be enforced. In a way, the notation has indeed been engraved (digitally speaking). Modifications can be made, but with much viscosity [19].

Taken together, these constraints impose a specific way of writing on composers that can adversely impact their creative process as it limits their capacity to experiment, to jot ideas down [31], and to iterate on the beautified symbols. The user experience is closer to writing on paper, but the freedom inherent to it is lost. EuterPen aims to offer composers the best of both worlds: reclaim part of that freedom while keeping enough structure to enable advanced manipulations of the music notation; and reclaim another part of that freedom to let composers externalize their ideas inside the score while keeping the main structure stable.

The first design and prototyping phase was driven by one high-level goal: to reveal the full potential of interactive surfaces for music score writing – prior work in the HCI literature suggesting that pen and touch input have the potential to resolve tensions between structure and flexibility. This phase (4 months) was primarily informed by the findings from Cavez et al. [12], who conducted interviews with nine composers about their creative process and how music notation programs support it.

Once developed, the first EuterPen prototype was used to illustrate a variety of possible interactions to two professional composers who accepted to participate in a half-day face-to-face workshop together with the authors. We believe it was important to make a high-fidelity prototype (see Section 3.2) available to participants even if we were still early in the design process, as composers are not necessarily very familiar with interactive surfaces. We wanted composers to be able to experience the new possibilities offered by this technology first-hand, enabling them to actually manipulate music notation with pen and touch. Discussions during the workshop were structured along different themes. First, the lead author gave an overview of EuterPen. Then, for each theme, a series of features and interaction techniques were showcased to the composers using video recordings, leading to discussions about the relevance of the feature and how it was mapped to interactions. EuterPen was running on a Microsoft Surface Studio 2 in the workshop room, and composers were encouraged to play with the system and try the features whenever they wanted. We made it clear to composers that despite the high fidelity of our prototype we were still at an early design stage, and that they should feel free to question and rethink what they were shown and propose new ideas during these thematic discussions.

As effective interaction design often lies in the careful attention to detail, the workshop then included a 1-hour brainstorming session focused on a specific operation to gather fine-grained insights. We chose copy-and-paste, an operation that is central to managing repetitions and variations in the music composition process, but often perceived by composers as tedious [12]. To avoid influencing composers during brainstorming, we did not present any solutions, but rather asked them to propose alternatives and improvements to the conventional Copy and Paste buttons. Workshop participants were invited to first produce around 10 ideas each, on separate post-it notes. After 15 minutes the resulting ideas were shared orally or by demonstrating how this would work in front of the EuterPen prototype.

The entire workshop was recorded (audio and video) and later transcribed by the lead author. A thematic coding [45] of those transcripts and recordings was performed by the first two authors.

Following this workshop, we entered a second design and prototyping phase (4 months) during which we developed new features and iterated upon existing ones based on the feedback from the two composers and new ideas that arose from the discussion. This led to the second version of EuterPen, which we used to validate our approach through interviews with a broader set of professional composers. Composers, as many creative professionals, are typically very busy, and we chose to conduct interviews online so that they could participate without traveling to our lab. We conducted 8 interviews, all performed using a videoconference tool. An interview lasted between 45 minutes and 2 hours 30 minutes (1 hour 40 minutes on average, more than 13 hours total). Information about the composers’ profile is available in Table 1. As we did for the workshop, the key features and interactions of EuterPen were recorded and grouped thematically into coherent feature sets. Those were made available to the interviewees on the Web, embedding the videos into a Web page together with textual explanations. That Web page was shared with the interviewees several days before the actual interview took place, so that they could familiarize themselves with the different features. The Web page was actually a form, that they could use to record early comments and ratings about individual features at their convenience. During the interview, this early feedback was revisited as part of the discussion between the interviewer and the composer. The lead author conducted the interviews, and the first two authors performed the transcription and thematic coding of those recordings. Ratings are reported by thematic group in Figures 3, 5 and 8.

Finally, we entered a third, shorter design and prototyping phase (1 month) to make final improvements to the prototype. These different stages constitute EuterPen's design journey, which we discuss in Sections 4.1–4.3, emphasizing the main insights from the different stages of the process.

EuterPen is developed as a Web application and can run in any browser that supports the W3C Pointer Events API [9]. It uses the VexFlow music notation API [13] to render scores as SVG elements [4], that can be imported from MusicXML documents with OpenSheetMusic [47]. The programmatic manipulation of the SVG-rendered music notation elements and all UI components is coded in JavaScript with D3 [7]. Handwritten music notation recognition is handled via a custom version of MyScript [30] configured with a specific grammar. This recognition service runs locally as a Java servlet that EuterPen queries to interpret digital-ink input.

EuterPen has been implemented to experiment with, and demonstrate, novel ways to support composers in their creative process. As such, we did not aim to develop a full-featured score editor but rather a functional prototype that can import music scores and let users modify them or write music from scratch. Section 4 discusses those novel ways to support composers, focusing on novelty and design choices rather than describing every single feature.

A musical piece may unfold temporally, but the creative process that yields it is most often non-linear. This is a source of tension as: on one hand, the notation program should help composers comply with the rules governing the piece's temporal – inherently linear – structure; but on the other hand, it should let them freely explore and arrange different ideas by supporting a very flexible, non-linear editing workflow.

Existing music notation programs are clearly focused on enforcing temporal-structure well-formedness, providing composers with a workspace organized strictly as a linear sequence of bars forming staves or systems of staves that all abide by the declared time signature. The constraints imposed by those programs do not apply to the primary music notation only, but prevent composers from inserting other pieces of information that are key to the composition process: text annotations or foreign objects such as images that are not meant to stay but support the creative process. Composers have to cope with these constraints, when an exploratory process rather calls for those constraints to be lifted, if only temporarily. When possible, composers will cheat the program: for instance, adding dummy bars at the end of a score to test ideas that are not meant to be part of the final piece. However, many constraints cannot be circumvented, leading composers to resort to other means to capture their ideas, such as pen and paper. This has a cost though, as they will then have to juggle completely disconnected media and workspaces.

Three actual photographs laid out horizontally and labelled (a), (b) and (c) show two images of two hands interacting on the screen and one image of one hand holding a stylus interacting on the screen. Image (a) shows the first finger of the right hand holding a stave and the first and second finger of the left hand ready to drag the stave below. Image (b) shows the finger of the right hand at the same place but the two fingers of the left hand lower, opening a pensieve between the staves. Image (c) shows a stylus performing drag-and-drop to put elements into this pensieve.

We address this global concern by designing EuterPen not as a regular score editor, but rather as a music processor.¹ While EuterPen has composers write music notation on staves as any regular score editor does, composers can interactively carve spaces between staves, that can hold a variety of contents. Two spaces coexist:

• the regular temporal composition space with a linear flow of staves, as found in all music notation programs;
  
• and canvases, named pensieves,² that composers can instantiate and populate freely with a variety of contents.
  

Pensieves can be used to store and retrieve any material relevant to the creative process (handwritten text annotations, drawings, pictures, audio samples and video clips). They can also be used to experiment with musical ideas, instantiating blank staves to write or copy music notation on – see Section 4.2. As such, pensieves exist in-between and around staves, and can be opened and closed at will.

Carving a pensieve between two staves in the temporal composition space involves holding one staff with a finger, and pulling the other staff with two fingers. This type of pensieve remains tightly integrated with the temporal composition space and scrolls along with it. A global pensieve can be shown to the side of the score as well by dragging it like a drawer. That global pensieve spans the full height of the EuterPen workspace and can be scrolled independently from the staves of the temporal composition space.

Pensieves let composers organize content around staves. But EuterPen also aims to give them more flexibility inside the staves themselves. The systematic enforcement of temporal constraints in bars represents a major source of frustration and impedes the creative process [12]. EuterPen addresses this key issue by relaxing these constraints. Staves can hold a mix of structured (engraved) spans and unstructured ones consisting of arbitrarily-long sequences of notes and rests. When writing music, composers can – but are not required to – draw bar lines, and then engrave or delete them at will. EuterPen also makes insertion more flexible in order to better support the creative process, which is highly non-linear, as mentioned before. A caret gesture [29] performed with the pen (see Figure 6-c) will push existing notes to make space that can then be used to write new notes. As the layout of bars on staves is something that composers consider important, the insertion strategy can be parameterized on-the-fly: adjust the amount of space depending on the (log-transformed) length of the caret gesture; after lifting the pen, choose to add space before (push content upstream) or after (push content downstream) the insertion point; and most importantly choose whether to automatically generate bar lines (which helps remain within a given duration) or not (which leaves the duration unconstrained).

EuterPen lets composers make further layout adjustments to engraved content, leveraging the structural information obtained in the process: spreading or packing notes with a two-finger pinch gesture, adding space to a bar by dragging the bar line with the pen. EuterPen does not attempt to optimize layout automatically as other score editors do, but rather gives composers the possibility to resize bars according to music notation rules on demand.

Beyond staves and pensieves, which constitute two distinct spaces with different purposes, EuterPen also lets composers make annotations anywhere in the workspace – in all areas, both temporal composition space and pensieves. Composers create those annotations seamlessly without the need to switch modes, as explained in Section 4.2.

A set of questions, photographs and diagrams laid out vertically. The questions are those asked to the eight composers during their interviews, and they are followed by their respective possible answers. The questions and answers are the following: "Q1: Do the actions shown in this video correspond to actions you would like to do when you compose?" (Yes/More or less/No), "Q2: Are you able to carry out this type of action with the tools currently at your disposal?" (Yes/More or less/No), and "Q3: Do you find what is shown in the video difficult to use?" (Very easy/Easy/Neither difficult nor easy/Difficult/Very difficult). Below, three photographs labelled "a) Work in a Free Space", "b) Localization of Thinking Spaces" and "c) Insertion and Manipulation of Bars" are laid out horizontally and illustrate EuterPen features that composers have seen during their interviews. The image on the left shows a handwritten bar and an engraved bar in a pensieve between staves. The image on the middle shows an empty pensieve on the left of the score and two hands opening an empty pensieve in the middle of the score. The image on the right shows a finger dragging a barline to push two bars on the right and make space in the bar on the left. Below each photograph, there is a 3-line diagram that reports participants’ answers to the three questions for the particular feature shown above. Each line diagram features 8 cells, one per participant. Cells are color-coded according to the participant's answer: green for Yes/Very easy, light green for Easy, yellow for More or less/Neither difficult nor easy, light red for Difficult and red for No/Very difficult. For all three features, the top line diagram features only green cells. The middle line diagram is mostly red for the first feature, features red and yellow for the second feature, and features multiple colors for the third feature. The bottom line diagrams mostly feature green and light green cells. Below the figure, a sentence explains how to interpret the middle line diagram: "Answers to Q2 color-coded red or yellow are to be interpreted in favor of EuterPen, as they indicate operations impossible or impractical in the score editors that interviewees currently use."

As illustrated in Figure 3, interviewees were very enthusiastic about EuterPen's way of breaking the score's linearity: canvases for free-form work and the flexibility in handling bars were unanimously judged to be useful (Q1). Interviewees did not think these exist in their current music notation programs, except for the insertion and manipulation of bars, which two composers felt they can already perform as they wish (Q2). Composers found the demonstrated interactions easy to use with the exception of one composer who found the caret gesture to insert measures potentially difficult to perform (Q3).

The idea of providing a blank canvas for composers to freely arrange and edit measures outside the score stemmed from the opportunities identified by Cavez et al. [12] and eventually grew into pensieves during the workshop. As C₂ put it: “it would be great to be able to copy, how shall I put it, copy and paste, but outside our score”, C₁ then suggesting this could be a “reserve of ideas, pensieve or clipboard”. The first EuterPen prototype that C₁ and C₂ saw during the workshop only allowed carving pensieves between staves. The concept was thus expanded to include the global pensieve (on the left in Figure 1), which is not tied to a particular place in the score but rather remains fixed on screen, serving as a global storage and annotation space.

The relevance of pensieves was further confirmed during interviews: “And yes, it leaves a little bit of... a little bit of a laboratory in the middle of the score” (BF). It is not just about “having everything already on screen rather than travelling around in windows,” but also about having space, “a bit like having cards and moving them around” (LA) instead of having to create extra bars, “which is hell, because then you have to take them away” (AO). Composers found multiple uses for pensieves. They can be simple temporary work areas, for example to “switch the left and right hands” (LA). Composers often saw them as a space to note comments, draw, but also to explore musical ideas, for example to “try out fifteen different rhythms” (LA). Staves created in pensieves (e.g. Figure 1 or Figure 2-c) are fully functional: they can be interpreted, played back, and “re-injected into the score at a later time” (DP), a feature that LA was enthusiastic about: “It's really incredible [...] adding staves on the fly underneath. Wow. Fantastic [...] You draw five lines by hand to make the poor man's staff, and now the computer does it itself. It's much more practical.”

Pensieves could even be used to build a structured library of ideas: “I can imagine this as a sort of idea box where you can have places where you have a zone with rhythmic ideas, a zone with ideas for a melodic sequence and you can take one or the other” (C2). This notion of a library extends beyond music symbols on staves to other creative resources: musical references or sources of inspiration (as in Figure 1), as C1 described during the workshop: “You can get the whole idea bit by bit. You don't have time to go into all the details of the thing, but then you have to note down as many resources as possible. If, for example, I want to take inspiration from Steve Reich, then I'll put Reich's score on the page so I don't have to go looking for it.” It can also be a teaching tool, as AO points out: “There were times when my teacher said to me: ‘Yes, but that, you have to look at Debussy, look at La Mer by Debussy. Look at all the string models’ ”. Or as OS says: “to make exercise sheets perhaps.” He sees pensieves as a tool to “import whatever you want. Images, text. Links to pages like that, [...] stories about music or anything else that might relate to the study of the orchestra, for example instruments in the orchestra.”

The greater flexibility obtained from the relaxation of structural constraints within measures was also received quite positively. Composers can experiment freely with melody and rhythm without having to worry about the notation's formal correctness, which fosters creativity: “There's no more alteration, there's no more rhythmic signature in fact? You can do what you like, you're free, aren't you? That's great. The example you did on the side, can you add beats, can you do a nine-beat bar like that, even without numbering it?” (YT), echoed by (BP) who observed that “the bar we're going to insert can be of any rhythmic signature” suggesting he would rather “define afterwards whether it's identical to the preceding bar or whether it's a completely different rhythmic signature.” BF further added: “There may be an irregularity at some point in the beat, a change of time signature, for example, or a change at the end of a phrase that runs counter to the beat”, which may lead to “the possibility of adding an extra beat to a bar”. Similarly, composers commented positively on the management of empty space in bars. While other programs automatically insert rests to keep a bar well-formed, EuterPen leaves composers free to write down their ideas unimpeded: “So typically, when you've deleted the two beats, it hasn't put anything in their place. I like that, that there's nothing there” (LA).

Composers also appreciated not having to worry about bar layout at the start of the writing process. As AO puts it: “when I'm composing, I'm spending time doing something and I, I shouldn't be spending time doing [layout]. In fact, ideally, I should be composing.” echoing observations from Bennett [5]. Layout stability – avoiding automatic optimizations – was also considered important: “Well yes, that's very practical because everything stays there. And then you do the layout again, but after you've finished the whole passage and not while you're adding notes one after the other” (LA).

To better match composers’ mental model and creative process, EuterPen supports two representations of the primary notation that afford different manipulations. The first representation uses regular, beautified symbols as found in other music notation programs. The second notation is the composer's own handwriting. The two can coexist seamlessly on a stave and even be intermingled in the same bar. The former representation, because of its engraved look & feel, suggests fully-developed, finalized material; while the latter rather suggests material that is still work-in-progress and more likely to change. This is further reinforced by the differences in how composers manipulate the two representations. Handwritten symbols can easily be deleted by flipping the pen and using the eraser – as on paper. On the contrary, engraved symbols must first be selected before they can be deleted. Composers can thus easily jot ideas down by drawing symbols on the score, and they can as easily discard them with the pen's eraser, without running the risk of inadvertently modifying or deleting engraved symbols nearby.

The two representations – engraved and handwritten – can be used on all staves, in both the main composition space and pensieves. One important design decision we made was to leave handwritten music symbols seemingly uninterpreted by default. Handwritten symbols necessarily have to be interpreted when composers ask to engrave them (Figure 4-f), but this interpretation actually takes place in the background even before that, as soon as they get selected (with a lasso selection gesture). This lets composers perform operations that would typically only be possible on engraved material on handwritten notation as well – while preserving their work-in-progress representation. Such operations include : copy-and-pasting the selected notes (Figure 4-b & c); listening to them (Figure 4-d);³ and searching for other occurrences of the same pattern throughout the score (Figure 4-e and Figure 5-a) – both engraved and handwritten. Finally, the conversion from handwritten to engraved notation is not a one-way transformation. Composers can actually toggle between the two representations at will (provided the selected symbols were actually input with the pen in the first place), which can be useful for instance to visually revert a passage to a more work-in-progress look & feel.

A set of two screen captures of EuterPen laid out horizontally. On the left, there is a selection of handwritten notes and a menu displayed above with six icons labelled from a to f. On the right, there is a selection of the same notes but engraved. The menu displayed above is the same except for the last icon, labelled f’, which is the reverse of the icon labelled f.

The decision not to interpret handwritten symbols immediately was taken for two main reasons. The first reason was to give composers as much freedom as possible, preventing the parsing process from forcing a particular way of writing on composers and thus avoiding the pitfalls of greedy input interpretation commonly found in other music notation programs [12]. The second reason was to enable three very different types of ink-based input to coexist without resorting to mode switches: handwritten music symbols; arbitrary secondary notations such as text and drawings (Figure 5-c); and a variety of gesture-based commands (see Section 4.3). Composers can seamlessly perform any of these thanks to the following disambiguation strategy:

• any pen mark that 1) has been initiated in empty space, and 2) does not match one of the three predefined gestures (Figure 6), will remain uninterpreted and treated as a simple annotation (secondary notation);
  
• any pen mark that has been initiated on a staff will be considered a candidate music symbol for interpretation, if selected later;
  
• if the trace currently being inked matches one of the predefined gestures, the ink color will change, providing feedforward to composers about how the gesture will be interpreted if they lift the pen then (for instance, the pin-to-play gesture in Figure 6-b);
  
• if this is not the interpretation they intended, composers can continue inking and lift the pen later when the gesture no longer matches any predefined one, which will be indicated by the ink reverting to its default color.
  

This strategy works all over the EuterPen workspace and lets composers add secondary notation not only in pensieves but anywhere on staves as well, more tightly integrated with the primary notation – for instance ink marks made to circle and link different passages on the score. Such secondary notation is actually not limited to handwritten text and drawings, but can be much more heterogeneous. Composers can drop different media types directly on the staves (replacing some bars) or very close to the staves. The list of supported media types is informed by results from Cavez et al.’s study [12] and by our workshop: an audio recording made on-the-spot using the tablet's microphone; a picture taken on-the-spot using its camera; a document (PDF, image, audio, video) imported from the file system or linked from a URL. The purpose and relevance of these secondary notations will evolve as the composition progresses. Composers can thus easily move, resize, show and hide, or entirely discard them at will.

A set of questions, photographs and diagrams laid out vertically. The questions are those asked to the eight composers during their interviews, and they are followed by their respective possible answers. The questions and answers are the following: "Q1: Do the actions shown in this video correspond to actions you would like to do when you compose?" (Yes/More or less/No), "Q2: Are you able to carry out this type of action with the tools currently at your disposal?" (Yes/More or less/No), and "Q3: Do you find what is shown in the video difficult to use?" (Very easy/Easy/Neither difficult nor easy/Difficult/Very difficult). Below, three photographs labelled "a) Music Handwriting and Conversion", "b) Cohabitation of Different Notations" and "c) Score Annotation" are laid out horizontally and illustrate EuterPen features that composers have seen during their interviews. The image on the left shows a selected handwritten sequence and a matching engraved sequence on another stave. The tip of a stylus is over an icon, ready to hide the search results. The image on the middle shows an audio sample and a photograph among engraved notation in the score. The image on the right shows hand holding a stylus, drawing annotations on the score. Below each photograph, there is a 3-line diagram that reports participants’ answers to the three questions for the particular feature shown above. Each line diagram features 8 cells, one per participant. Cells are color-coded according to the participant's answer: green for Yes/Very easy, light green for Easy, yellow for More or less/Neither difficult nor easy, light red for Difficult and red for No/Very difficult. For all three features, the top line diagram features mostly green cells. The middle line diagram features only red cells for the first feature, features mostly red cells for the second feature, and features multiple colors for the third feature. The bottom line diagrams mostly feature green and light green cells. Below the figure, a sentence explains how to interpret the middle line diagram: "Answers to Q2 color-coded red or yellow are to be interpreted in favor of EuterPen, as they indicate operations impossible or impractical in the score editors that interviewees currently use."

As illustrated in Figure 5, interviewees were very positive about EuterPen's way of breaking the score's homogeneity: mix handwritten and engraved notations, insert other media such as audio samples on staves, put secondary notations anywhere, were possibilities deemed useful by a majority of composers (Q1). Those features are largely absent from current music notation programs (Q2), and seemed generally easy-to-use in EuterPen (Q3).

Secondary notations are useful not only to make text annotations but to make freeform drawings as well – what LA calls “supervised cohabitation” – that can capture intent, help understand something, give information about musical structure (see Figure 1): “Maderna's Serenade pour un satellite is a circular score”...“musically [...] it's illegible” but “in a circle you can make choices, you can go left and right... [so] the idea of engraving it like that rather than explaining it logically, was really so that it would inspire musicians” (LA). Drawing could also be a way to “invent symbols that you're going to put in the score” (AN) or to support musical ensembles as a way to “build the score by inserting an idea of physical movement, even explaining it with a diagram, showing everyone's place, with a movement that's taken as you would on a sketch” (YT).

The idea of integrating music passages in forms other than that of the primary notation (an audio sample, a picture of a roughly sketched passage to be transcribed - see Figure 5-b) had emerged during the workshop. Its relevance was further confirmed during the interviews: “It would be fantastic if... [the system could play] what was engraved, written, and given that there's the MP3 afterwards, given that there's the audio afterwards, we'd really like it to follow on” (LA). Similarly, C₁ wished they could work with audio recordings of themselves searching for ideas on the instrument and integrate those easily in the score: “I have a lot of memos in my phone that should be developed, but they are in my phone, and I cannot put them in the score.” Several interviewees saw potential in the combination of pensieves and secondary notations as a means to notate and coordinate group work: “It can be really useful because it transforms the score into a performance” (DP). On the same topic YT said: “We're in the process of doing something, creating a show with the double bass orchestra. We're making it by hand. In other words, there are lots of things. It starts with an idea. And then the work in progress happens in rehearsal, with five people, and everyone puts in their two cents.” And since “the sixth person wasn't there because she's away on business”, so it's like “an instruction manual” for her.

The possibility to make the handwritten and engraved representations coexist had much success.⁴ Composers appreciated that many operations could be performed on handwritten passages as well, such as play them, copy-and-paste them, or look for other occurrences (Figure 4): “Searching for the motif to see if you've already written it before beautifying it, I'd use that every day” (LA). During the workshop, both C₁ and C₂ commented very positively on the possibility to immediately listen to handwritten passages using a simple gesture (Figure 6-b) without having to engrave them. But at a more fundamental level, the possibility to switch back-and-forth between engraved and handwritten representations was seen as a means for composers to keep track of where they are in their creative process: “You don't have to ask yourself the question ‘Is this a final version, is this my draft?’ ” (LA), because this enables “visualiz[-ing] very, very easily a passage that needs to be reworked” (DP). As C₁ observed, it could also help composers keep track of the decisions made: “Going back in time can really help. Often you get the clean version and can't remember how you got there.” But during the workshop composers also wondered how this would work for engraved notation not originally written by them (for instance, bars imported from a MusicXML file). C₁ emphasized that it would be “seriously awkward to see the handwriting of someone else”. In such cases, one option discussed with C₁ would be to use machine learning to train EuterPen on the composer's own handwriting and to have it generate the handwritten representation, possibly involving ink beautification and handwriting generation mechanisms [43].

While some score editors do enable the annotation of scores, composers saw the possibility to draw and add pictures or audio recordings directly on the staves as a means to support creativity, that could be useful for teaching as well: “We can even, while keeping what [the student] has written, add something else in manuscript [...] something that can go in another direction” (YT).

As discussed in section 4.2, one of EuterPen's core design principles is to make multiple types of ink-based input coexist without resorting to mode switches: handwritten music symbols, secondary notation, and predefined gesture-based commands. In addition to these, and consistent with current editors, EuterPen should support pen- and touch-based direct manipulation of music notation elements. This introduces further ambiguities that need to be resolved.

Predefined gesture commands. Three actions are triggered by pen-based gestures that are easy to learn and to perform (Figure 6). These gestures are recognized by a simple, robust classification algorithm that accounts for variability in gesturing. To be scale-independent, it uses the gap between two staff lines as a reference distance to classify input traces: lasso-to-select encloses an area at least 2 · gap wide and high, ending within 2 · gap of the start point; pin-to-play combines a lasso with a vertical trace at least 4 · gap high; and finally caret-to-insert is a downward vertical trace of at least 4 · gap high followed by an upward trace.

A set of three actual photographs of EuterPen labelled (a), (b) and (c) and laid out horizontally. The image (a) shows the tip of a stylus drawing a lasso in blue. the image (b) shows a hand holding a stylus and drawing a pin in green. the image (c) shows a hand holding a stylus and drawing a V-sign in orange. A circle with "+9" in it is displayed next to the tip of the stylus.

Direct Manipulation. Music notation elements that lie on staves can be moved horizontally or vertically to change the pitch or adjust the layout. For instance, when selecting a note head, a small cross will appear (see Figure 8-c), suggesting the possibility of direct manipulation in these directions. Dragging along another direction will initiate one of different lasso selections, as detailed next.

The digital pen is an excellent tool to write music symbols. But it is also a very precise selection tool, that can be used to delineate free-form areas. As such it is well-suited to the selection of music elements, which are composed of multiple tiny glyphs densely packed together. The modern staff notation is a multi-dimensional grammar that encodes different auditory attributes of a note (pitch, duration, etc.) by combining those glyphs and positioning them precisely on staves and ledges. The most frequent glyphs include the head, stem, flag, beam, dots and accidentals. Some glyphs define a note's auditory attribute, while other glyphs are rather modifiers of that attribute. But regular score editors typically consider notes as entities that cannot be further decomposed. Selecting any constituent glyph of a note will select the whole note, that can thus only be moved (adjusting pitch), copy-&-pasted, or deleted.

EuterPen aims to break this monolithic view on notes and let composers select individual auditory attributes of a note if they want to. This is made possible by introducing novel types of selections. Composers can invoke those novel selection tools seamlessly, still without switching modes. We achieved this by adopting a strategy that disambiguates what to select based on where composers start their selection. A lasso enclosing multiple notes will select:

• the whole notation if initiated outside of any glyph – see Figure 6-a;
  
• handwritten notation only if initiated on handwritten symbols, ignoring engraved notation;
  
• a series of durations if initiated on a rhythm-related glyph (beam, stem, flag or rest) – see Figure 8-a;
  
• a series of pitches if initiated on a note head, dragging diagonally;⁵
  
• accidentals if initiated on an accidental;
  
• text if initiated on textual elements such as, e.g., dynamics.
  

To help composers perform those different selections, EuterPen provides feedforward, dimming the glyphs that are not related to one of the selected auditory attributes until the lasso selection is completed. Figure 8-a illustrates this feedforward for a selection initiated on a rhythm-related glyph (in this case a quaver), temporarily dimming note heads and accidentals.

While selecting individual auditory attributes makes little sense for coarse manipulations such as deletion, it provides composers with a whole new set of capabilities for more elaborate actions such as Find or Copy-&-Paste.

A set of two photographs of EuterPen labelled (a) and (b) laid out horizontally. The image (a) displays an example of the search function with several melodic sequences highlighted in yellow. Handwritten annotations are also present on the score. A minimap is displayed on the right of the score. The image (b) is a zoom on the minimap of image (a), showing the original selection as a blue circle and the matching sequences as yellow circles. The annotations also appear on the minimap.

Find. Repetitions and variations are an essential part of many compositions [12]. Highlighting the occurrences of a melodic pattern is thus an important feature of music notation programs. EuterPen highlights such occurrences directly in the score, and also shows them on a minimap of the score (Figure 7-b), facilitating navigation over the result-set. Composers can then effortlessly add these occurrences to the active selection and manipulate them together – for instance to adjust their relative pitch. Combining this feature with EuterPen's novel selection capabilities actually enables composers to make queries involving specific auditory attributes only. They are able to search for a rhythmic pattern regardless of pitch or, as illustrated in Figure 7-a, search for a pitch sequence regardless of rhythm.

Copy-&-Paste. Music has always been about repetition and variation (as AO says: “Mozart would have been so happy to be able to copy and paste notes! I think he must have copied quite a few...”), and composers copy-&-paste music notation very frequently. Available in many score editors (Figure 8-b, Q2), the possibility to duplicate music notation this way represents a major advantage of computer-based editing over paper. But the way regular score editors implement copy-&-paste is very monolithic and provides little feedforward. It makes it difficult for composers to anticipate what will happen, and sometimes yields unexpected or unwanted results. EuterPen takes a different approach to copy-&-paste. Dragging a pen-made selection with a finger will duplicate it (drag-to-duplicate). While dragging, notation currently at the destination will fade-out almost completely to show composers where the duplicate notation will be placed if they drop at this moment, as illustrated in Figure 8-b. Another way to control precisely where to paste is to explicitly select what to replace (select-to-replace). Once a selection has been copied, composers can select what symbols they want to replace with what is in the system's clipboard.⁶ A tap on the PASTE button in the selection contextual menu (Figure 4-c) will then effectively replace the notation. Again, combining this feature with EuterPen's novel selection capabilities opens new possibilities. By restricting the selection to one auditory attribute only (pitch, duration), composers can for instance copy a sequence of pitches and paste them on another rhythm than the original one; or conversely they can copy a rhythm and paste it on another pitch sequence than the original one – all without having to erase and rewrite the entire sequence.

A set of questions, photographs and diagrams laid out vertically. The questions are those asked to the eight composers during their interviews, and they are followed by their respective possible answers. The questions and answers are the following: "Q1: Do the actions shown in this video correspond to actions you would like to do when you compose?" (Yes/More or less/No), "Q2: Are you able to carry out this type of action with the tools currently at your disposal?" (Yes/More or less/No), and "Q3: Do you find what is shown in the video difficult to use?" (Very easy/Easy/Neither difficult nor easy/Difficult/Very difficult). Below, three photographs labelled "a) Selection of Musical Elements", "b) Copy and Paste of Musical Elements" and "c) Manipulation of Musical Elements" are laid out horizontally and illustrate EuterPen features that composers have seen during their interviews. The image on the left shows a hand holding a stylus and drawing a lasso to select a rythmic sequence. The image on the middle shows a finger dragging a musical sequence to duplicate it in another bar. The image on the right shows the tip of a stylus on a selected note. A cross is displayed under the tip of the stylus to show the directions in which the note can be moved. Below each photograph, there is a 3-line diagram that reports participants’ answers to the three questions for the particular feature shown above. Each line diagram features 8 cells, one per participant. Cells are color-coded according to the participant's answer: green for Yes/Very easy, light green for Easy, yellow for More or less/Neither difficult nor easy, light red for Difficult and red for No/Very difficult. For all three features, the top line diagram features mostly green cells. The middle line diagram is mostly red for the first feature, features multiple colors for the second feature, and is mostly yellow for the third feature. The bottom line diagrams mostly feature green and light green cells. Below the figure, a sentence explains how to interpret the middle line diagram: "Answers to Q2 color-coded red or yellow are to be interpreted in favor of EuterPen, as they indicate operations impossible or impractical in the score editors that interviewees currently use."

As illustrated in Figure 8, interviewees were very enthusiastic with EuterPen's innovative way of breaking down musical elements to select, copy-&-paste and manipulate them with precision. Composers were almost unanimous in recognising these features as useful (Q1), although half of them have already found workarounds with their current music notation programs (Q2). They found the way EuterPen supports these features easy to use (Q3).

Selection plays a key role in many operations and is of major concern to composers. Interviewees found EuterPen's extended set of selection tools useful (Figure 8-a) and shared their thoughts about how they would use them in their work. For instance, OS would like to select melodic voices with ease “to erase and replace with something else because I made a mistake. It's often the intermediate voices in particular that I have problems with. It's true that you often have to erase everything and start all over again.” YT saw potential in auditory attribute selections, for instance to try different rhythms for a given pitch sequence: “If we have a melody that we like, we can change the rhythm. I've sometimes started with a sequence of notes, and thought to myself ‘what can I do with this? Here, I'd like it to have this groove.’” But also the other way around, to manipulate only the melody: “We'll change the pitches and keep a form that's close by changing the notes [...] I'll take this melody. So that will be my wink and reference to that melody, but I'm going to reverse it.”

Combining search with auditory attribute selections to highlight full but also partial matches for a given melodic pattern was also commented positively about. Modern classical composers seek structural clarity and our interviewees talked about how they could use relaxed queries constraining rhythm only: “When you're doing an orchestration with lots and lots of content, it can be nice to say ‘Wait, let's see all the people who do this rhythm’” (AO). Similarly, for melodic patterns: “[...] you search for the subject at the beginning and see it pop up in the score, and that's it. Your analysis is done. Even hidden motifs will appear, things that can be overlooked because they're placed between voices two and three, they'll appear on their own” (LA). DP and BF, both jazz composers, give even more importance to “series of notes” and “series of rhythms”: “I only work with small pieces of material that I repeat and transform, and which end up in different forms, sometimes very hidden in different parts of the score” (DP). BF explains how he would use the search feature to find a formula to edit: “I know that I've put this formula in the middle of a development and I want to find it again. Maybe it's quicker that way, since it's highlighted. So it provides a focused reading of the big picture.” Composers including AN, LA and BF also expressed the need to extend the search's scope: to be able to search in pensieves and to include secondary notation as well. The minimap (Figure 7) was first suggested during the workshop, and requested again by several interviewees, but only added to EuterPen in the last prototyping phase. It shows not only melodic pattern matches during a search, but bar lines and handwritten annotations as well, as these provide strong navigation cues.

Feedback about direct manipulation of the primary notation was positive (Figure 8-c). BF emphasized the need for fine-grain manipulation to improve legibility: “when the voices are very, very close together, simply to avoid blots. And if you have accidentals in addition to this crowded writing... it gives you clarity.” While most composers can already adjust symbol placement with their current editor, some complained about the rigidity of such manipulations and their lack of stability: “It moves all the notes. For example, I often have a problem with arpeggio notes that, you know, immediately change the dimensions of the score. Just by having this arpeggio, you end up with a bar that's the size of two bars. And then, typically, every time I want to move something, it's going to move the whole score” (AO). Interviewees also shared ideas about how to further improve direct manipulation of the primary notation, as briefly discussed in Section 6.

During the workshop's brainstorming session, C₁ and C₂ both imagined that sequences of notes could be dragged outside of the score, and dwelled on the concept of a rich-yet-straightforward manipulation of specific auditory attributes, either when copying or when pasting. This inspired our alternative proposals to classic copy-&-paste, which were well received by interviewees. Commenting about drag-to-duplicate, AO observed: “I like the fact that you can already perform an action without having to click on a button,” BF adding: “This movement, we did not have this ease with the editors. It had to be copied... Moving is complex because you often have to cancel a measure to put something back in [...]. [In the meantime you need to put] rests, because an empty measure is not tolerated by the software.” Interviewees described different uses of copy-&-paste combined with auditory-attribute selections. Leveraging pensieves, AN saw potential for “a sort of idea box where we have a zone with rhythmic ideas, a zone with melodic progression ideas and we can take one or the other.” LA, who insists on the importance of stability of the score, further stresses the importance of spaces where he can work on duplicated elements before integrating them into the main temporal flow: “It's a good thing that the pseudo-final material, or the engraved version in any case, is relatively fixed, so that we don't tamper with it too much, but rather copy [some of] it into draft spaces and work there. That way, there's a version that moves less, otherwise, in fact, we'd be, I think, very tempted to throw everything left and right. [Things] would be moving all the time.” Other composers (BF, CI and OS) also expressed interest for auditory attribute copy-&-paste, for instance to change the rhythm of a melodic sequence, acknowledging how easy EuterPen makes such operations: “This is something we do regularly when there are repetitions of formulas [...] except that we often take only the rhythm. We take the formula and then edit the notes one by one, on the formula we've just copied. So it's a lot more laborious” (BF).