Multi-Modal LLM Assisted Visualization of Large Visualization Collections

Visualization as a field helps researchers make sense of large volumes of information, but requires careful decision making and design considerations. Rather than assume that individuals become expert visualization practitioners in their limited free time, institutions sometimes employ a service provider to consult on the visualization process. In most cases, the service provider consists of only a small team or individual, yet is expected to support the visualization needs of the entire university.

Two of the authors of this article are in this position and can speak about some of the primary challenges. Consultation requests, for instance, can come from researchers representing diverse disciplines, which can vary widely not only among themselves but also from the service provider's own domain expertise. It can therefore be challenging to discern what visualization standards already exist in the researcher's field. Consulting the relevant literature is helpful but becomes a significant time investment just to first understand the field's prevalent visual encodings. Furthermore, when determining where to focus our efforts, we lack visibility into which types of visualizations and tools are most commonly used across campus, and by extension, which areas most urgently need support. These are not insurmountable issues, but they can constrain a service provider's impact. Our project, Vis-Sieve, seeks to address these obstacles by allowing visualization service providers to quickly explore the wide range of visualizations produced across their institution. The project starts by gathering open-access publications from an institution's researchers, then extracting the figures from the publications, adding labels, and finally presenting the figures and associated labels in an interactive web interface. The overarching goal is to give service providers a more efficient way to view visualization trends in unfamiliar domains, while also providing support for informed facility development, trend tracking, technique discovery/development, service catalog entry inspiration, and advanced practitioner discovery and recruitment.

Because this project is still a work in progress, we would like to focus on a specific sub-task within this project: the automatic labeling of figures by chart type (e.g., bar chart, heatmap, scatter plot). Our implementation builds upon existing work within the AI4Vis realm by using OpenAI's ChatGPT-gpt-4o-2024-08-06 model to classify visualizations. Once labeled by the model, these figures become filterable within the system by chart type, enabling users to explore specific categories of visualization quickly and effectively. This becomes a powerful tool for insight building, and time saving in the hands of a visualization support service provider.

Our work was divided into three phases: 1) data collection by extracting pdfs and their figures, 2) labeling the extracted figures, and 3) visualizing the data for exploration.

T

The Vis-Sieve project is still in development, but it has already made significant progress in enabling the exploration of a wide variety of visualizations produced at an institution. Although several steps could use refinement, the classification stage shows promising accuracy and saves researchers many hours of manual effort.

To evaluate the robustness of our classification pipeline, we retrieved random samples of figures and compared the labels generated by GPT-4o-mini's model to first, our own manual labels and second, to labels assigned by the public VisImages dataset [8]. For the comparison to manual labels, we conducted three rounds of random sampling on the dataset (100 images per round), and found an average manual verification accuracy of 91.2%. Encouraged by these initial results, we then used the public VisImages dataset for a more direct comparison. In the first test, many VisImages samples contained multiple sub-visualizations, posing challenges for a direct one-to-one label match. As a result, we adopted a partial matching approach: if a label assigned by both VisImages and zero-shot annotation was present in the image, it was considered a match, and only images that had no shared label were deemed unmatched. This resulted in approximately 65.2%–81.2% matches for labeled cases, but the methodology was imperfect due to the inherently multi-label nature of some figures. For example, the extracted visualization in Figure 3 could be labeled with bar chart, line chart, map, or heatmap/2D-density-plot labels, and all would be valid.

T

To refine the comparison, we isolated each sub-visualization within a figure before classification, thus ensuring a one-to-one mapping between an image region and a chart label. Across 35,016 isolated sub-visualizations, we observed 20,539 matches and 14,477 mismatches (58.64% matches). Mismatches did not necessarily imply a clear error: in some cases, labels diverged due to different, yet valid, interpretations of the same visual element.

T

To probe this further, we selected 100 of these isolated images that produced mismatches for manual inspection. Table 1 shows that VisImages and zero-shot annotation both labeled 56 images correctly; 33 were correct only by zero-shot annotation, 8 only by VisImages, and 3 were labeled incorrectly by both methods, resulting in an overall accuracy of 89%. We also discovered examples where both labels were different but arguably correct. Figure 4 read as a scatter plot by VisImages but interpreted as a table by zero-shot annotation when viewed at a broader scale—underscoring the subjective aspects of visualization classification.

Cost and scalability between manual and automated labeling were also evaluated, with automated labeling coming out as the clear winner. Labeling 35,016 images cost approximately $32.69 in API usage. A single script processed 5–8 images per minute; parallelization could lower runtime. Manual labeling at 5 images per minute would require about 116.72 labor hours and cost roughly $2,334.40 (assuming $20/hour). This price difference favors automated classification for institutions needing to label tens of thousands of figures rapidly and cost-effectively.

Our findings highlight both the benefits and inherent complexities of automating multifaceted visual content labeling, and although our application pipeline effectively processes collections and supports analysis, some steps yield diminishing returns, prompting us to explore methods to alleviate these compounded limitations.

This article presents the progress made in the Vis Sieve project, which builds a system that provides publication retrieval, visualization extraction, automated analysis, and subsequent visualization to support small teams of visualization service providers for large research institutions. When it comes to labeling visualizations, the use of zero-shot classification highlights the potential for minimizing manual annotation efforts, making this method particularly valuable for institutions seeking scalable solutions for visualization analysis. This work also showcases a successful cross institution collaboration with student engagement at its core.

In the future we hope to expand our visualization interface to support a presentation of the image collection in 3D using similarity scores between images. This will involve creating Vision Transformer embeddings for each visualization. Following embedding, we will project high-dimensional vectors into lower-dimensional space for exploration. Using t-SNE and UMAP, we preserved both local and global relationships, allowing clusters of similar visualization types to emerge [14][15]. This representation of our visualization collection may allow for more rapid exploration without requiring a categorical taxonomy to be provided at all. We welcome collaborators on this project; if you're interested, please reach out to Devin or Carolina.