Shuoqi Chen
When Poetry Touches Painting
What happens when a poem jumps from text to sight, and steals the paint brush?
When Poetry Touches Painting Short Demo
This work turns a verse into a sequence of visual edit prompts, letting poetry take the director’s chair and steer the image in real time. The poem is revealed line by line. For each line, a few keywords are selected as visual anchors, and each keyword triggers a localized edit that transforms the canvas again and again. The editing is performed through prompt driven image modification (Nano Banana Pro), with a bounding box marking the intended region of change so the viewer can see where the word is acting on the image.
For each line, a few keywords are selected as visual anchors, and each keyword triggers a localized edit that transforms the canvas
Two accompanying metrics investigate the transformation as a kind of instrument panel. Edit uses DINOv2 features to estimate how much the visual content shifts between frames, independent of the text. Align uses CLIP to estimate whether the direction of the change conforms to the active keyword. Both are mapped to a 0–100 range and displayed as filling squares as the edit completes. In the background, multiple computation modules perform analyses of the text-to-image synergy, including foreground/background segmentation, depth analysis, and guided prompt design.
A sequence of prompts targets local regions of the image, gradually transforming the scene toward the imagery suggested by each line of poetry. DINOv2 features track visual changes across edits, while CLIP scores measure the alignment between each localized transformation and the guiding text.
Artistically, the piece treats poetry as a set of cues that gently tug an image in new directions, one keyword at a time, giving words the power to paint a story. As each edit lands, the original scene slowly yields to the poem’s imagination while still pretending to be the same picture. It poses a playful but serious question: when does an image stop illustrating its source and start illustrating the poem, and who is really steering that shift: the painting’s original intent, the model’s visual instincts, or the words of the poetry insisting on one more change?
Safety-critical AI by day; art and imagination at heart.
Shuoqi Chen is a machine learning and computer vision engineer at Intuitive Surgical, where he develops AI-driven imaging technologies to support lung cancer diagnosis. He studied robotics at Carnegie Mellon University, and before that he studied at Dartmouth College, where he served as director of design at Dartmouth Inc. and held various design roles at DALI Lab. By day, he works on safety-critical AI; At night, he chases art and imagination.