Organizing online hackathons for newcomers to a scientific community – Lessons learned from two events

Jeaime Powell, The University of Texas at Austin, United States, jeaimehp@gmail.com
Linda Bailey Hayden, NSF Science Gateway Community Institute, United States, haydenl@mindspring.com
Amy Cannon, Omnibond Systems, LLC, United States, amycannon@omnibond.com
Boyd Wilson, Omnibond Systems, LLC, United States, boyd@omnibond.com
Alexander Nolte, University of Tartu, Estonia, alexander.nolte@ut.ee

Acquiring computing skills is essential not only to work in computer science but also because many impactful discoveries occur at the interface between traditional scientific disciplines and computing and data science. Time-bounded events such as hackathons can provide an opportunity for newcomers to experience programming firsthand in a collaborative environment. Just providing access to computational resources, however, is not sufficient because newcomers will likely require guidance and support in order for them to perceive their participation in a hackathon as a positive experience propelling them to future success in the subject. We have developed a hackathon format for this purpose that we have successfully applied during in-person events for two years. The global pandemic of 2020, however, forced us to move towards a virtual format. In this paper, we report on our experience making this transition. We will specifically elaborate on two online events, discuss successes and failures and provide suggestions for hackathon organizers.

CCS Concepts:Human-centered computing → Empirical studies in collaborative and social computing;

Keywords: online hackathon, community engagement, learning, time-bounded collaborative event

ACM Reference Format:
Jeaime Powell, Linda Bailey Hayden, Amy Cannon, Boyd Wilson, and Alexander Nolte. 2021. Organizing online hackathons for newcomers to a scientific community – Lessons learned from two events. In Sixth Annual International Conference on Game Jams, Hackathons, and Game Creation Events (ICGJ 2021), August 2, 2021, Montreal, Canada. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3472688.3472700

1 INTRODUCTION

Many impactful discoveries occur at the interface between traditional scientific disciplines such as chemistry, geology, biology and astronomy, and computer science. Acquiring skills in the latter areas is thus not only crucial for individuals that aim for a career in a computer science-related field but arguably also for those that aim to work in other disciplines. Starting to acquire computing skills by, e.g., enrolling in courses can, however, appear intimidating, especially for individuals with limited prior exposure.

Time-bounded events such as hackathons can provide an opportunity to get exposed to, experience, and learn about computing first hand [5, 11, 15]. During such events, participants form teams and engage in intense collaboration over a short period of time to complete a project of their interest [12, 13]. Since their beginnings in the early 2000s hackathons have become a popular form of collaboration [4, 17] and have been adopted in (higher) education [5, 15], (online) communities [2, 6, 17], entrepreneurship [3, 10], corporations [8, 13, 14], and others. Providing access to computing resources, asking teams of newcomers to work on projects of their choice, and offering on-demand technical support as is common in many hackathons [1, 16] cannot be expected to be sufficient in this context, though. Instead, newcomers will require guidance not only related to solving technical issues but also related to choosing, scoping, and executing a project [11]. It is also essential to create a welcoming atmosphere for newcomers since some hackathon formats have been criticized for fostering a competitive climate [18] that favors individuals that already possess technical expertise [7].

Over the past three years, we have developed and refined a hackathon approach that aims to foster newcomer engagement with high-performance computing (HPC). Starting in 2018, we have organized annual in-person hackathons in conjunction with two of the main conferences in the field1. The global pandemic of 2020, however, forced us to move towards a virtual format. In this paper, we report on our experience making this transition.

Based on our experiences organizing two online hackathons in the summer and fall of 2020, we have identified potential pitfalls that organizers, mentors, and participating teams should be aware of and provide suggestions on how they can be addressed. We believe that these insights can be valuable for other organizers that aim to run online events specifically for newcomers to a scientific community.

2 TWO ONLINE HACKATHONS

In this section, we will describe how we organized the two aforementioned online hackathons. We had a team of five organizers – who are also the co-authors of this paper – that started preparations for both of the 4-day hackathons about six months before each event (Figure 1 shows impressions of the second hackathon). We will organize the following description based on key activities and decisions that we needed to take before, during, and after each event [9, 12]. These included participant and mentor recruitment (section 2.1), general event preparation (section 2.2), activities at the beginning of the event like ideation and team formation (2.3) as well as activities during the event including setting the event agenda (section 2.4), mentoring (section 2.1), prizes (section 2.7) and stakeholder involvement (section 2.8). Moreover, we will also elaborate on the tools that organizers, mentors, and participants used during the event (section 2.6). We will also outline changes that we implemented for the second event.

Figure 1
Figure 1: Participants during the fall 2020 online hackathon.

2.1 Participant and mentor recruitment

We started preparations for each hackathon by identifying and inviting individuals from our networks within the HCP community that might be willing to serve as mentors during the event. We initially focused on individuals who had prior teaching experience and background in different scientific disciplines, including chemistry, geography, biology, and others. The invitation we sent to them included basic information about the event, such as a preliminary schedule and information about their role. This role would include each mentor outlining a rough project idea, theme, or problem description, presenting it to the participants of the respective hackathon, and guiding a team during the event. In parallel, the organizers started contacting colleges and research institutions in their networks to recruit student participants. Students received an invitation that included a link to a Google Form which they could use to register. The form also included questions related to basic demographic information as well as the student's addresses so that we could send them swag (section 2.7). In total, we managed to recruit mentors and participants for the first and mentors and participants for the second hackathon.

2.2 Specialized preparation

To ensure that students could work with technologies from the HPC domain during the hackathon, we organized Google Cloud Credits before each event. We also created a Github page that provided core information for participants, including the event schedule, links to tools we utilized for communication during the event (section 2.6), links to tutorials, and contact information of mentors and organizers. We also prepared and ran two separate webinars during the week before each hackathon. For the first webinar, we only invited the mentors, and we briefed them about our expectations towards their role, including how they would engage with the participants before and during the hackathon (section 2.5). During the second webinar – for which we only invited the students – we ran multiple tutorials on how to use Github and Google Cloud to prepare them for the upcoming event. Both webinars were recorded and shared on the event's Github page for future reference.

2.3 Ideation and team formation

For our hackathons, we followed a two-stage approach towards ideation and team formation. We first asked mentors to prepare project ideas, themes, or problem descriptions before each hackathon (section. The mentors then presented these to the students during the kick-off meetings. Each kick-off lasted about 2 hours and was held via Zoom. Afterward, we created separate Zoom breakout rooms for each mentor. Students could then join each room to find the mentor and project theme they were most interested in. We took this approach to mimic the way we organized ideation and team formation during the in-person events we had previously organized. Before the students left to join the different breakout rooms, we told them that they needed to decide on a project before leaving Zoom altogether to ensure that each student had a team. After each student had found their respective breakout room, the students and mentor in each room engaged in an ideation process to develop the concrete project idea that the respective student team would be working on during the hackathon. They were free to utilize any ideation approach that they deemed suitable. We instructed mentors to let students find a project idea themselves and only provide input related to the technical complexity of the different project ideas that the students came up with.

Leaving the students to decide for themselves which team to join, however, led to a situation where one mentor only had one interested student while another mentor had 10. In the interest of fairness – each team should have roughly the same size – we suggested for the single student to join another team and the large team to split up into two teams of roughly the same size. We told both teams that they could work on the same theme suggested by the mentor but had to develop their own project idea. To be able to support the second team, we recruited another mentor on the spot.

Our ideation and team formation approach was generally successful in that teams were formed and were able to develop an idea they could work on during the event. After the hackathon, some participants mentioned though that they felt pressured to decide on the spot which project to join without having the opportunity to explore their options and discuss with mentors and other fellow students. For the second event, we thus modified our approach slightly. Instead of asking the students to decide on the spot and not leave Zoom before deciding on a team, we held the kick-off event one day before the hackathon. The students then had until the next morning to discuss with their peers and with the mentors to decide for which team to join and which project to attempt.

2.4 Agenda

In addition to the kick-off event during which we conducted the ideation and team formation (section 2.3), explained the event agenda, outlined the code of conduct, explained the judging criteria (section 2.7) and provided information about how to contact organizers and mentors (section 2.6), we also held checkpoints in the morning and evening of each hackathon day. The checkpoints were attended by all organizers, mentors, and students and lasted for about 2 hours. For each checkpoint, we asked teams to prepare a short overview of their progress, describe challenges they had faced, and outline their plans until the next checkpoint. The exact details of what teams should present were announced during the previous checkpoint. After each presentation organizers, mentors, and other teams had the opportunity to provide suggestions and feedback to the presenting team. We also engaged teams in different social challenges, such as designing a team background for Zoom, choosing a team song, and recording short videos where teams engaged in various distance challenges. Each checkpoint also featured a short talk by one of the event sponsors and a technical webinar. The topics of these webinars – which were held by the organizers – were decided based on suggestions by mentors or students.

In addition to the joint checkpoints, we also planned mentor-only meetings to discuss each team's progress, identify potential issues, and provide suggestions to the mentors based on our prior experience. During the first of the two online hackathons, we planned these meetings to take place right after each joint checkpoint. This approach ended up not being feasible, though, because teams wanted to meet with their mentors directly after each checkpoint to discuss the feedback they had received and plan future steps. For the second event we thus organized the mentor-only meetings once in the middle between the 2 daily joint checkpoints.

Teams presented their projects during the final checkpoint, which was streamed live so that conference participants and other members of the HPC community could join (section 2.8).

2.5 Mentoring

The mentors played a crucial role in our approach. They were responsible for providing a project idea, theme, or problem description as a basis for the teams to develop their own project idea. They also conducted the ideation process (section 2.3) and were available during the event to answer questions and provide feedback. In addition, they were our primary source of information about how teams were progressing apart from the checkpoints (section 2.4).

While this approach worked well, we noticed during the first event that one mentor per team might not be sufficient. Since mentors were also community members and were as such involved in the conferences that ran in parallel to the hackathon, they were not always able to provide timely feedback, address issues, and generally devote as much time as was sometimes necessary. For the second hackathon, we thus recruited additional mentors to have two mentors per team, which also provided the opportunity for less experienced mentors to participate in and support the event.

2.6 Tools suggested and tools used

We mainly utilized Slack, Zoom, and Github for the hackathon. Each event had its own Github page, which served as the main information hub (section 2.2) during the event. We utilized Zoom for all synchronous meetings (kick-off, checkpoints, and final presentation) and Slack for asynchronous communication in between. In Slack, we created one general announcements channel to, e.g., send reminders for upcoming checkpoints and one channel where teams could ask for feedback and help. All organizers, mentors, and participants were invited to both channels. In addition, we also created one channel where we only invited organizers and mentors to discuss potential organizational issues that arose during the event, e.g., related to teams potentially requiring specific support.

We also created one channel per team to which we invited the respective mentor and student participants of that team. While we suggested that teams utilize that channel during the hackathon, most teams only used it to get in touch with their mentor and had other tools to communicate with each other instead. These included Discord, Facebook Messenger, iMessage, Teams, and others.

After the first hackathon, some students mentioned that they had utilized their own Zoom subscriptions or subscriptions of friends and colleagues to stay in touch with their team members or to quickly organize a video meeting among the team members if necessary. For the second event, we made sure to secure subscriptions for each team to utilize during the event so that they would not have to organize this themselves.

2.7 Prizes

Both hackathons were organized as competitive events during which teams could win a jury prize and a viewers’ choice prize. The jury prize was given based on judging criteria which were announced during the kick-off (section 2.4). The criteria included aspects such as the project's viability and usefulness, its technical complexity, and the final presentation's quality. The jury consisted of members of the HPC community (section 2.8). The viewers’ choice prize was given based on votes from the broader public, including members of the HPC community. The link to the voting form was announced during the final presentation session.

We also sent a swag package to each participant before the event containing stickers, dice, a headset and other things.

2.8 Stakeholder involvement

We involved members of the broader HPC community in different ways. We recruited them as mentors and judges, we invited them to the final presentation, and we provided the opportunity for them to vote in the viewers’ choice awards. We also made sure that winning teams would be recognized during the awards ceremonies of both conferences to further spread the word about the teams and their projects in the broader HPC community.

3 POTENTIAL PITFALLS AND LESSONS LEARNED

While organizing the two aforementioned events, we identified pitfalls and learned valuable lessons. We will discuss them in the following and provide suggestions on how to address them.

  • Online events require additional planning and structure: Before the online hackathons, we invested more time for preparation than for our previous in-person events by creating an information hub, organizing different means of communication, and getting in touch with mentors and participants. This preparation still proved to be insufficient. This became especially evident during the team formation process (section 2.3), which we had typically organized in an ad-hoc way during our in-person events. It became clear, though, that the students required more time and scaffolding during the online event. One approach to address this issue could be to ask students to form teams before the event, develop an idea, and register together as a team.
  • It is harder to stay in touch with teams: During in-person events, it is possible to walk around in the room(s) that the hackathon takes place in, talk to teams and mentors and thus retain an overview of how the event is going and where additional support might be needed. In an online setting, the primary contact points with teams are pre-planned meetings such as checkpoints. This can severely limit the possibility for organizers to spot issues. In order to mitigate this issue, it is essential to carefully select and brief mentors because they are in closer contact with the teams. Thus, they can point organizers towards issues and ask for support if the teams themselves refrain from doing so themselves. It might also be advisable for multiple mentors to support the same teams since individual mentors might not be able to continuously stay in touch with their team due to parallel duties.
  • Be aware of side activities: During our online hackathons, we observed multiple instances where students and mentors had parallel commitments related to their job, studies, the conference that ran parallel to the hackathon, and their private life. This affected most teams and led to additional organizational overhead. Moreover, teams often failed to sufficiently communicate parallel commitments in time, which led to team members having to take on additional tasks to, e.g., be able to present something during the scheduled checkpoints. Teams did not break up subsequently, but parallel commitments certainly caused disruptions. During an in-person event, such activities would likely be easier to communicate and occur less frequently. In order to address potential issues, mentors and students should talk about parallel commitments so that everyone can adjust in time.
  • Participants might disengage: We never had such a case during our in-person hackathons and only one during the online events we report on in this paper, but participant disengagement still appears to be easier in an online than an in-person setting. There probably is not much that organizers can do to prevent this from happening, but everyone (organizers, mentors, and participants) should be aware if someone appears disinterested and actively engage with that individual to prevent her/him from dropping out.
  • Teams might have different collaboration styles: Teams exhibited different collaboration styles during the two online hackathons. Some remained on video calls most of the time, others mainly worked in parallel and only synchronized occasionally, e.g., before checkpoints. Some teams also split into subgroups or switched between styles. Apart from being aware of this, organizers and mentors must stay in touch with teams to ensure that they remain in communication with each other and that every participant is on board.
  • Teams might use different tools: As organizers, we aimed to provide an environment where participants did not have to think about which tools to use for collaboration. They could simply utilize the Slack channels we had created before the event and the joint Zoom room. Many teams, however, decided to utilize their own tools. Despite this, we still perceive it as beneficial to provide a solid technical basis for team collaboration even if teams should decide not to use it. Organizers should, in any case, make sure that participants know how to reach them, the mentors, and their team fellow members. For this, we suggest a combination of a joint knowledge base – we utilized a Github page as discussed before – where mentor and organizer contact information can be found and a detailed briefing during the kick-off meeting.
  • Technical issues can become magnified: Our participants were primarily newcomers to the HPC community with limited or no prior experience. It was thus sometimes difficult for them to understand and deal with the plethora of different tools and platforms they encountered during the hackathon. Solving technical issues online can be much more complicated than in an in-person setting. Thus, it is thus advisable to plan for more support staff than in an in-person setting. In addition, it should also be noted that participants might have technical issues unrelated to HPC tools, such as an inadequate internet connection or power outages that in an in-person event would either affect most participants or none. While it is not possible for organizers to address such issues, it might be advisable to provide additional means of reaching organizers and mentors, e.g., via phone, so that participants can get in touch to find a solution.

ACKNOWLEDGMENTS

The authors gratefully acknowledge support by Omnibond Inc., Science Gateways Community Institute Workforce Development (funded by the National Science Foundation under award number ACI-1547611), Elizabeth City State University, Texas Advanced Computing Center, Globus, Intel, and Google. We would also like to thank the hackathons mentors and participants like Tiara who is a wonderful example of the many students whose internships tie back to the hackathons.

REFERENCES

  • Nataly Birbeck, Shaun Lawson, Kellie Morrissey, Tim Rapley, and Patrick Olivier. 2017. Self Harmony: rethinking hackathons to design and critique digital technologies for those affected by self-harm. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 146–157.
  • Ben Busby, August Matthew Lesko, et al. 2016. Closing gaps between open software and public data in a hackathon setting: user-centered software prototyping. F1000Research 5(2016).
  • David Cobham, Kevin Jacques, Carl Gowan, Jack Laurel, Scott Ringham, et al. 2017. From appfest to entrepreneurs: using a hackathon event to seed a university student-led enterprise. In 11th annual International Technology, Education and Development Conference.
  • Jeanette Falk Olesen and Kim Halskov. 2020. 10 Years of Research With and On Hackathons. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 1073–1088.
  • Kiev Gama, Breno Alencar, Filipe Calegario, André Neves, and Pedro Alessio. 2018. A Hackathon Methodology for Undergraduate Course Projects. In 2018 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9.
  • Daniela Huppenkothen, Anthony Arendt, David W Hogg, Karthik Ram, Jacob T VanderPlas, and Ariel Rokem. 2018. Hack weeks as a model for data science education and collaboration. Proceedings of the National Academy of Sciences 115, 36(2018), 8872–8877.
  • Lilly Irani. 2015. Hackathons and the making of entrepreneurial citizenship. Science, Technology, & Human Values 40, 5 (2015), 799–824.
  • Marko Komssi, Danielle Pichlis, Mikko Raatikainen, Klas Kindström, and Janne Järvinen. 2015. What are Hackathons for?IEEE Software 32, 5 (2015), 60–67.
  • Maria Angelica Medina Angarita and Alexander Nolte. 2020. What do we know about hackathon outcomes and how to support them? - A systematic literature review. In Collaboration Technologies and Social Computing. Springer.
  • Alexander Nolte. 2019. Touched by the Hackathon: a study on the connection between Hackathon participants and start-up founders. In Proceedings of the 2nd ACM SIGSOFT International Workshop on Software-Intensive Business: Start-ups, Platforms, and Ecosystems. 31–36.
  • Alexander Nolte, Linda Bailey Hayden, and James D Herbsleb. 2020. How to Support Newcomers in Scientific Hackathons-An Action Research Study on Expert Mentoring. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1(2020), 1–23.
  • Alexander Nolte, Ei Pa Pa Pe-Than, Abasi-amefon Obot Affia, Chalalai Chaihirunkarn, Anna Filippova, Arun Kalyanasundaram, Maria Angelica Medina Angarita, Erik Trainer, and James D Herbsleb. 2020. How to organize a hackathon–A planning kit. arXiv preprint arXiv:2008.08025(2020).
  • Ei Pa Pa Pe-Than, Alexander Nolte, Anna Filippova, Christian Bird, Steve Scallen, and James D Herbsleb. 2019. Designing Corporate Hackathons With a Purpose: The Future of Software Development. IEEE Software 36, 1 (2019), 15–22.
  • Ei Pa Pa Pe-Than, Alexander Nolte, Anna Filippova, Chris Bird, Steve Scallen, and James D. Herbsleb. 2020. Corporate Hackathons, How and Why? A Multiple Case Study of Motivation, Projects Proposal and Selection, Goal Setting, Coordination, and Outcomes. Human-Computer Interaction(2020).
  • Jari Porras, Antti Knutas, Jouni Ikonen, Ari Happonen, Jayden Khakurel, and Antti Herala. 2019. Code camps and hackathons in education-literature review and lessons learned. In Proceedings of the 52nd Hawaii International Conference on System Sciences.
  • Porya Mohajer Soltani, Kalevi Pessi, Karin Ahlin, and Ida Wernered. 2014. Hackathon: A method for digital innovative success: A comparative descriptive study. In Proceedings of the 8th European Conference on IS Management and Evaluation. 367–373.
  • Nick Taylor and Loraine Clarke. 2018. Everybody's Hacking: Participation and the Mainstreaming of Hackathons. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 172.
  • Jeremy Warner and Philip J Guo. 2017. Hack. edu: Examining how college hackathons are perceived by student attendees and non-attendees. In Proceedings of the 2017 ACM Conference on International Computing Education Research. 254–262.

FOOTNOTE

1Please refer to the following URL for an overview of the past hackathons we organized in this context: http://hackhpc.org/

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

ICGJ 2021, August 02, 2021, Montreal, Canada

© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8417-9/21/08…$15.00.
DOI: https://doi.org/10.1145/3472688.3472700