Myungin Lee & Noah Bissell & Ethan Paley & Amanda Wang
Sensorium Arc: AI Agent System for Oceanic Data Exploration and Interactive Eco-Art
Sensorium Arc (AI reflects on climate) is a real-time multimodal interactive AI agent system that personifies the ocean as a poetic speaker and guides users through immersive explorations of complex marine data. Built on a modular multi-agent system and retrieval-augmented large language model (LLM) framework, Sensorium enables natural spoken conversations with AI agents that embodies the ocean’s perspective, generating responses that blend scientific insight with ecological poetics. Through keyword detection and semantic parsing, the system dynamically triggers data visualizations and audiovisual playback based on time, location, and thematic cues drawn from the dialogue. Developed in collaboration with the Center for the Study of the Force Majeure and inspired by the eco-aesthetic philosophy of Newton Harrison, Sensorium Arc reimagines ocean data not as an abstract dataset but as a living narrative. By translating scientific data into an emotionally resonant, conversational experience, the project aims to spark curiosity and effectively communicate complex climate knowledge to the general public. The project demonstrates the potential of conversational AI agents to mediate affective, intuitive access to high-dimensional environmental data and proposes a new paradigm for human-machine-ecosystem.
Introduction video
Overall diagram of the Sensorium Arc system
Input Processing begins when a sensor detects a visitor's proximity, triggering the system. Microphone input passes through active noise cancelling and speech-to-text conversion to produce a clean User Query.
LLM Pipeline handles the query along two paths. The Visualization Agent reads from a database of visual descriptions and selects appropriate imagery, outputting a Visual Selection Token. Meanwhile, the Query Rewriter + Retrieval Agent reformulates the query and pulls relevant Retrieved Documents. The Responder Agent then combines the user query, the visual selection, and the retrieved documents to generate a Response.
Response Processing routes the output to the right channels: the visual selection feeds a Visuals Map, while the text response is processed by a Keyword Matcher and Text-to-speech.
Audio-Visual Layers render the final experience. The Visuals Map drives pre-rendered videos, globe data overlays, and the base globe view, while the keyword matcher triggers synchronized audio-visual events like subtitles, UI changes, and camera positioning, and text-to-speech delivers the spoken answer.
a) Exemplary setup for the Sensorium Arc exhibition Users engage through a natural gesture, whispering into a seashell as if speaking to the ocean. b) User interface before activation showing the sensors c) Activated user interface showing the response from the system
a) Default glove view before user interaction. b) Chlorophyll concentration data visualized on the globe surface. c) Water clarity rendered using layers of hyper spectral diffuse attenuation coefficient (Kd) from NASA's PACE satellite (2024). d) Combined visualisation of atmospheric CO2 levels and ocean surface wind flow, highlighting interconnected environment.
Audio-visual layers are directly tied to the user's interactive experience, combining data visualization, event triggers, video playback, localization, and text-to-speech (TTS) audio accompanied by synchronized subtitles for accessibility.
Our modular globe visualization database include sea surface temperature, ocean currents, atmospheric CO2 emissions, chlorophyll concentration, water clarity and more from diverse sources including NASA Earthdata and earthaccess.
Narrative-focused videos courtesy of Virtual Planet visualize sea-level rise, plastic waste dispersion, ocean acidification, and overfishing.
Visuals courtesy of Virtual Planet Technologies
Myungin Lee is a researcher and artist designing multimodal XR experiences grounded in HCI, DSP, and machine learning. He earned a Ph.D. in Media Arts and Technology from UC Santa Barbara and M.S. and B.S. degrees in ECE from Hanyang University. He is affiliated with the AlloSphere, designing large-scale interactive immersive systems. He is a faculty member in the Immersive Media Design (IMD) program and Computer Science at the University of Maryland, College Park. His work has been featured at Ars Electronica, the Kennedy Center, Getty PST, NeurIPS, IEEE VR, IEEE VIS, ACM CHI, and ACM SIGGRAPH.
Ethan Paley is an undergraduate student pursuing dual Bachelor of Science degrees in Computer Science and Immersive Media Design at the University of Maryland. Their academic interests include interactive entertainment development, sustainability, and the intersection between art and technology.
Noah Bissell is a programmer and aspiring technical artist, currently pursuing a B.S. in Immersive Media Design (IMD) and Computer Science at the University of Maryland. His work as a student of the IMD program, exploring areas from data visualization to gestural interfaces, have been showcased at campus events and the Kennedy Center.
Amanda Wang is a new media artist with a Bachelor of Arts in Immersive Media Design and a minor in Sustainability Studies from the University of Maryland, College Park. Her work explores interactive installations, generative art, and real-time visual systems, with a focus on the combination of technology, ecological responsibility, and embodied interaction.