Colin: A Multimodal Human-AI Co-Creation Storytelling System to Support children's Multi-Level Narrative Skills

Digital storytelling systems now commonly integrate AI[49] to encompass both the act of telling a story through narration and the depiction of the story's events in visual form. OSOS[30] leverages AI technology by profiling a child's language environment with pervasive language modeling, extracting priority words for learning, and generating bespoke storybooks that incorporate these words[30]. The digital tool Communics[41] is utilized to facilitate children in collaborative comic-based digital storytelling. These systems also empower children to actively participate in and influence story development through decision-making[11], thereby providing more opportunities for storytelling. Interactive storytelling has been demonstrated as an effective educational method [10, 15, 22, 31]. It has been proven that it can improve children's language and narrative skills [5, 34, 50, 51, 54, 56]. However, these studies focus on how to improve the performance of interactive storytelling system, but do not pay attention to the learning process of children's interaction with the story system. With the emergence of large language models, recently, some studies have incorporated large language models into children's interactive storytelling activities. Nisha uses automated planning in natural language text generation to create stories that are coherent and believable [12, 13, 44, 45]. The qualitative research of Yuling Sun proves that parents have a need for interactive story systems [46].

Previous studies incorporate LLMs into educational technology [29]. Generation methods include virtual reality avatars, voice interface creation stories [7], through drawing creation, tangible interface [14, 33, 55] and graphic interface [35, 43]. Such tools highlight the significance of multi-modal creative support, allowing children to express themselves through various mediums. However, few studies have addressed the core understanding of children's grasp of stories through participatory creation. Benton presents a storytelling system for narrative co-creation through a game [9], Zarei came up with a enactment-scaffolded narrative authoring tools [36]. But these studies do not integrate the educational goals and understanding processes of stories. Some studies have noted that interactive systems improve children's math [40, 53], coding skill [16, 17, 28, 39] and narrative skills [32]. These adaptations demonstrate the application of LLMs in creating educational content [18, 21], improving student engagement and interaction [3, 6, 47], and providing personalized learning experiences [24, 42]. LLMs have been used to generate children's narratives [4, 20, 23]. In different research trajectories, some scholars have used LLMs to create intelligent learning partners that are able to collaborate with humans [26], provide feedback [27], and encourage students [19, 47]. A common application involves employing LLMs as dialogue partners in written or oral form, such as in the context of task-oriented dialogues that provide opportunities for language practice [19, 52], or as story continuation partners to collaborate with children in creating stories [20, 37].

In contrast to previous studies, Colin incorporates novel design features aimed at enhancing children's visual comprehension and narrative association skills. While the systems previously discussed are predicated on aiding children in the articulation of narrative richness, Colin, through a model that interlinks pictorial elements, encourages children to contemplate and expand upon discrete elements, thereby cultivating the completeness and connectivity within narrative skills.

To generate the design principles of the participatory story interactive system for children, we recruited five education experts and three parents to conduct semi-structured interviews. Each participant had more than three years of experience in educating or raising children. These interviews focused on three points: educational purposes, benefits of interactive storytelling for children, and problems that facilitator currently encounter in interactive storytelling. We summarized the experts’ viewpoints and derived the following three design goals in table 1.

Colin uses multi-modal LLMs to help children expand on the contributions they wish to add, and effectively weaves the key ideas and knowledge points into the development of the storyline. It has five interactive steps in interactive storytelling process: 1) Question Guidance: The system generates open-ended questions to elicit narrative contributions from children; 2) Storyline Expansion: Based on children's feedback, the system further develops the storyline to support continued narrative development; 3) Story Element Completion: After children complete some story elements through drawing, the system assists them in completing the story content; 4) Key idea Questions: The system generates questions related to the key ideas of the story, based on children's participation in the creation process, to assess their understanding of the central idea; 5) Explanation of key ideas: The system will affirm children's answers and further explain the key ideas expressed in the story to help them deeply understand the moral and knowledge points of the story. Figure  2 shows the overview of the Colin process.

A round of dialogue introduces the story and provides background information. The Colin asks the children questions and allows them to decide the story's progression at this point.

To accommodate children's short attention spans, limiting questions to two may help maintain focus. At the story's end, the AI poses additional questions and clarifies reasoning. Conversations can occur in multiple rounds; if a child's response contradicts key ideas, the AI provides guidance. Children interact with Colin by pressing a voice button, which lights up during recording. Pressing it again stops the recording, and Colin continues the story based on the child's input while upholding key ideas. After contributing, children can access a drawing board to illustrate elements of their creation. Upon story completion, Colin asks two questions to assess understanding of knowledge and values. Children can engage in multiple voice interactions, and even incorrect responses prompt Colin to patiently explain key ideas. In the final assessment, children use the drawing tool to create or modify elements and manipulate story components, aiming to craft a new story that conveys the original values.

Colin is a web-based app implemented in the React based on JavaScript. For the backend server, we implemented on Flask with Python. We adopted GPT-4 to generate text. Besides, we use ChatGLM [8] to generate images. Colin consists of five key interaction processes 3.

Human-feedback Preview In story generation, parents first input the texts that children use. Then Colin uses outline generation prompt(Appendix A.1.1) to combine the story outline with the character and keywords selected by parents to generate a new story outline. After obtaining the story outline, we use the few shot method to hint the LLM to split the story correctly(Appendix A.1.2), and then the LLM divided the story into multiple parts. Once the outline and story fragments are generated, parents can choose to manually re-generate them to ensure the quality of the story text. Then Colin will extract story background and character description through extract prompt(Appendix A.1.3, A.1.4). We use the ChatGLM[8] API to generate background pictures and character pictures from extracted description information. Parents can click on the unsatisfactory existing image to regenerate images. Through LLM and human feedback, we get story text fragments and images used in story interaction.

Story Co-Creation In the story co-creation session, the story fragments are expanded into a complete plot of each chapter through the story generation prompt (Appendix A.1.5), and the LLM is prompted to generate interactive questions in the middle of the chapter to realize the children's ability to co-create the story plot.We save the conversation context in whole process to ensure LLM achieve consistency in storytelling in later story chapters. Colin uses interface provided by iFlytek[25] to implement story text to voice and children voice to text transformation. After the child clicks on the speak button, the device turns on the built-in microphone for recording. When the child clicks on the speak button again, the audio file is generated and sent to the back-end voice module for parsing into response text and for continuing the story. The child's response will be used to build feedback to generate an extended continuation of the story for the next chapter (Appendix A.1.6). Colin also provides the ability to draw sketches by offering the drawing board function based on p5.js[2], providing a variety of color brushes for children to draw.

Story Connection Build Once co-creation is complete, Colin allows children to express their understanding of the story by clicking a corresponding button in the navigation bar. The system gathers character images and sketches from previous chapters, providing a blank drawing board for children to illustrate their interpretations. They can place image elements onto the board and use a brush tool to enhance their drawings.

Evaluation Metrics As shown in Table 4, we designed evaluation metrics including two modules: text generation and question generation. The text generation module is divided into plot extension text and truth extension text, and the question generation module is divided into plot question and truth question. In this method, evaluators were shown stories where one is generated by GPT-4 and another one is generated by a human. They were asked to choose from each pair based on the evaluation dimension. We asked three evaluators to choose from each pair of generated text based on the different dimensions given in the index system. All assessments were conducted by three university student research assistants with more than one year of experience in the education of children.

Results As the figure 4 demonstrates, GPT-4 matches human-generated content in intelligibility and surpasses it in logical rationality, attraction and overall preference.

The evaluation result shows that human-generated questions scored lower on story relevance, possibly due to participants associating questions with various life situations rather than focusing on the story. GPT-4 generated questions were more inspiring and inclusive, better promoting story development and gathering meaningful feedback from children. Human participants sometimes created questions more related to daily issues or educational goals rather than the story's core content, often resulting in yes/no questions that limited children's response opportunities. In contrast, GPT-4’s questions were more closely aligned with the story's plot and encouraged more comprehensive answers from children. Therefore, GPT-4 can assist in expanding and training children's thinking skills in the field of children's storytelling education, making it suitable for deployment in interactive storytelling systems.

We recruited 20 child participants (11 female, 9 male, aged 5 to 13 years old, M=7.97, SD=2.81) from child training institutions and kindergartens through offline recruitment. This experiment was approved by the Institutional Review Board (IRB) of the affiliated university. Each participant received a 20CNY compensation for the 60-minute session.

The pre-test phase aimed to assess children's understanding of the story‘s knowledge before interacting with Colin. Before the experiment, a demographic questionnaire was administered to the children to collect basic information. During the pretest phase, the children were asked questions about the fundamental knowledge in the story.

At the beginning of the experiment, the experimenter introduced each participant on how to use the system and provided 5 minutes for self-practice and inquiries. Participants utilized Colin through the experimenter's browser on their personal computers, with their computer interfaces being recorded. Parents first read the text, selecting the main theme they wished to convey through the story. After reading the text and visually assisted story content, children were required to narrate a story with a similar theme but different plot, based on elements randomly generated by the system. They were also expected to create informational and logical associations between the elements provided by the system. During the children's narration, the experimenter recorded the children's recognition and use of the system's prompted elements (characters or scenes) to assess their ability to map and link visual information. Additionally, the time interval from when the children viewed all the elements to the start of their narration, as well as the proportion of pauses and self-correction time during the narration process, were recorded. These metrics were used to evaluate their capacity to process and match the acquired visual information. Finally, the children's expressed content was analyzed for the use of conjunctions and coherence of sentences to assess their handling of logical associations between matched elements. After completing the story narration, participants were asked to fill out a questionnaire assessing their level of engagement during the interaction with the system. Subsequently, they were invited to take part in a 20-minute semi-structured interview regarding their user experience and usability of Colin.

To evaluate how Colin influenced understanding and narrative skill improvement process, we referenced the Language Arts definition in "U.S. Common Core State Standards" [1], and narrative metrics in previous research [38] to evaluate narrative skills through storytelling. We evaluate three level narrative skill in four dimensions, knowledge, internal response, key ideas, and narrator evaluation through a pre-test and post-test questions scored.

The three different levels in the evaluation are:

• Level-1: Children identify characters in the story, assigning the basic knowledge of characters in the story.
  
• Level-2: Children can understand the internal response of characters and the key ideas of the whole story in their own expression.
  
• Level-3: Children express their own story, transferring elements from the original story, and narrate to justify why an action or event took place.
  

To prevent children from merely repeating their pre-test answers in the post-test, the wording of the post-test questions was slightly modified. A comprehensive list of sample questions can be found in the appendix 3.

To measure children's enjoyment in interacting with artificial intelligence or humans, we adapted a questionnaire consisting of 4 items based on the prior work of Waytz [48]. The experimenters verbally presented questions to the participants and guided them to indicate their level of agreement using a graphical Likert scale. The 5-point scale was visualized as five circles, each depicting a different facial expression: "Strong Agree", "Agree", " Neither Disagree Nor Agree", "Disagree", and "Strong Disagree". The circles varied in size, with the smallest representing the least happiness (crying face) and the largest representing the greatest happiness (laughing face). Researchers pointed to each circle when describing the corresponding option. The Cronbach's alpha for internal consistency was 0.77, indicating acceptable reliability.