AI Reverse Art: A Digital Journey Deconstructing Van Gogh's "Starry Night Over the Rhône"

The University of Washington's Innovative Breakthrough AI technology has once again pushed boundaries, this time "deconstructing" Van Gogh's masterpiece!

With just an original painting as input, artificial intelligence can mimic Van Gogh's brushstrokes, gradually recreating the entire painting process.

Comparing the two sides closely, the AI has achieved an almost 1:1 replication, showcasing the construction process of the entire image.

This remarkable technology, named "Inverse Painting," originates from the University of Washington's laboratory. The project's academic achievements have been recognized and will be featured at the prestigious SIGGRAPH Asia 2024 conference. Notably, the research team includes two Chinese scholars: Bowei Chen from Northeastern University and Yifan Wang, a graduate of ShanghaiTech University, who have contributed to this groundbreaking research.

Delving into Technical Details

How does this technology work? Behind it lies a complex system that uses a diffusion model-based inverse painting method to transform input images into artworks. This process involves several main stages:

Learning the painting process: The project team collected 294 acrylic painting videos, which, after meticulous processing, allowed the AI to learn real artists' creative techniques.

Generating painting instructions: The AI analyzes different elements (such as sky, trees, and figures) and their relationships, then generates a series of instructions telling itself what to paint next.

Diffusion model image generation: Using these instructions, the AI can gradually add details, simulating the human artist's painting process, building the complete work bit by bit.

Text and region understanding: Besides generating instructions, the AI also creates region masks, ensuring each element of the painting is in the correct position, thus improving the accuracy and effect of the painting.

Progressive rendering and time control: The AI starts with a blank canvas, gradually generating a complete painting sequence while mimicking the time intervals of real creation, giving the entire painting a more natural and fluid feel.

Training and Testing Process

The entire process can be divided into two main phases:

Phase One: Instruction Generation

This phase deploys two intelligent generators: one for text instructions and another for region selection. The text instruction generator compares the target work with the current canvas, offering specific suggestions like "paint the sky" or "add flowers." Meanwhile, the region selection generator creates a binary image, precisely marking areas that need modification. The combination of these two instructions ensures the AI can paint accurately in the appropriate locations.

Phase Two: Canvas Rendering

The canvas update phase puts the previous stage's instructions into practice. It employs a diffusion model-based rendering technique that gradually refines clear images from noise. During the update process, the renderer considers multiple factors, including text instructions, region selection, time progression, and feature comparison between the target work and the current canvas. This multi-dimensional consideration allows the AI to more closely approximate the creative process and style of human artists.

Testing Phase

In the testing phase, the AI system demonstrated its ability to create complete paintings from scratch. Two core features are worth noting:

Coherence: The AI uses a self-referential method, with each stroke building upon the previous one, ensuring the coherence and logic of the entire creative process, much like a human artist carefully considering each brushstroke.

Time simulation: The system sets fixed time intervals between each canvas update, cleverly simulating the passage of time in real painting, making the AI's creative rhythm closer to that of humans.

Ultimately, compared to three baseline methods (Timecraft, Paint Transformer, Stable Video Diffusion), its generated results were significantly superior.

Community Discussion and Controversy

Once published on Reddit, this technology immediately sparked widespread discussion. The most popular comments expressed concern for the artist community.

However, many people also held positive attitudes. They believe this technology could become a powerful tool for learning painting, providing valuable learning resources for art enthusiasts. Software like XXAI can offer excellent painting ideas for art lovers.

Conclusion

In summary, the University of Washington's "Inverse Painting" project vividly demonstrates the wonderful combination of technology and art, while also prompting people to think. How can we maintain unique human creative perspectives while using AI to enhance creativity? How do we find a balance between technology and art? These are topics we need to explore deeply as we embrace this exciting technological future.