Study Notes on Agentic Visualization: Extracting Agent-based Design Patterns from Visualization Systems

Autonomous agents powered by Large Language Models (LLMs) are reshaping the field of Artificial Intelligence (AI) and visualization, necessitating an examination of how these changes affect human agency in analytical processes.
The emphasis on human-centered reasoning in visualization raises questions about the autonomy, delegation, and coordination of these agents that contribute to analytical tasks.
This paper introduces a concept termed agentic visualization (AV), describing systems that include independent AI agents while preserving human agency in analysis.
Design patterns for agentic visualization are extracted, focusing on roles, communication, and coordination.
Contributions: - Introduction of the AV framework for designing and analyzing agentic systems that maintain human-centered analytical approaches. - A collection of reusable design patterns for agentic visualization derived from existing literature on automated visualization. - A demonstration of applying these patterns to a real-world scenario, illustrating how they can create a dynamic visualization dashboard.

Definition of Agents: - Agents are defined as software entities capable of perceiving their environment and taking goal-directed actions, embodying agency that spans from simple reactions to complete autonomy.
Agentic Visualization (AV): - Description: The integration of agents with interactive visual analysis systems that balances computational autonomy with human analytical control. - AV extends paradigms of visual analytics by enhancing agent autonomy beyond reactive tasks. Unlike traditional systems, AV allows agents to proactively identify analytical opportunities and suggest or manage tasks independently. - This shift raises questions about idle agent behavior — whether agents should remain passive or proactively explore data without user commands.
Mixed-Initiative Interaction: - AV builds on mixed-initiative systems that facilitate human-computer collaboration by allowing both humans and agents to actively engage in analytic processes. - The focus is on agents' capacity to make autonomous decisions and adapt their approaches based on intermediate results, improving the overall analytic process without continuous human direction.

The evolution of agent technology, from early reactive agents to current autonomous systems powered by LLMs, reflects an ongoing shift in capabilities, while confronting challenges such as poor generalization and scalability.
Ethical considerations and questions of user trust in agent systems persist, emphasizing the need for design patterns that guide effective human-agent collaboration.

Patterns were identified through a thorough analysis of visualization systems incorporating human-agent collaboration across prior research, focusing on systems with semiautonomous or autonomous components.
The identification is framed around four criteria: direct interaction, emphasis on data analysis, use of interactive visualization, and inclusion of autonomous elements.

Agent Role Patterns: Responsibilities and scopes of agent actions within visualization.
Communication Patterns: Structuring the exchange of information between agents and users, maintaining transparency and facilitating feedback.
Coordination Patterns: Managing workflow dynamics between agent and user activities, regulating when and how actions occur.

Roles include patterns such as Forager, Analyst, Chart Creator, and Storyteller.
Communication patterns include Insight Timeline, Progress Indicator, and Provenance Log.
Coordination patterns consist of Scouting, Swarming, Monitoring, and Consolidating.

Forager: Agents responsible for parallel exploration of data based on human-defined parameters. Agents collaborate to retrieve and report findings, exemplified in systems like SurfaceMapper.
Analyst: Agents that aid human users—particularly data analysts—by suggesting initial analysis strategies and adapting recommendations based on user interactions. Systems like Jupybara demonstrate this pattern effectively.
Chart Creator: Agents assisting in design and refinement of visualizations, aimed at reducing workload through automated processes. Implementations include DYNAVIS and Data Formulator.
Storyteller: Agents that automate storytelling by combining text and visuals into narratives, an enhancement seen in systems like DATAWEAVER and Data Director.

Insight Timeline: A shared channel for humans and agents to communicate findings and coordinate efforts without redundancy, as seen in InsightsFeed and DataSite.
Progress Indicator: Provides task progress updates to enhance user understanding of analysis stages, demonstrated in ProReveal and InsightsFeed.
Provenance Log: Keeps track of data transformations and sources to ensure reproducibility and trust in the analytical process, seen in VisTrails.

Scouting: Multiple agents explore datasets to pinpoint areas of interest, fostering autonomy while allowing user guidance, illustrated in Mix&Match and DataSite.
Swarming: Multiple agents collaborate towards shared goals, dividing tasks but synchronizing efforts effectively. Examples include Saka et al.'s clustering algorithm.
Monitoring: Automated tracking of ongoing data streams to alert users of significant changes without overwhelming them, with applications in InsightsFeed and ProReveal.
Consolidating: Agents serve to compile and integrate insights from various analytic paths, proving useful in systems like InsightLens and DataDirector.

The application of AV patterns to a scenario involving medical research highlights issues regarding literature saturation and rapid publication growth. Challenges address query formulation, scaling, and the synthesis of findings across various studies.
Agents effectively assist in refining search methodologies, monitoring incoming publications, and leveraging AV patterns like Forager, Scouting, and Storyteller to manage complex workflows.

As AV systems gain traction in visual analysis, exploring the balance of agency and control becomes crucial, challenging traditional paradigms of human-computer interaction.
Ethical implications regarding transparency, accountability, and the interpretability of AI-driven decisions must be addressed continually, fostering a reliable partnership between humans and agents.
Future work should focus on enhancing the sophistication of agents while ensuring that user insight remains at the forefront of the analytical process.