Not to be outdone by Meta’s Make-A-Video, Google today detailed its work on Imagen Video, an AI system that can generate video clips given a text prompt (e.g., “a teddy bear washing dishes”). While the results aren’t perfect — the looping clips the system generates tend to have artifacts and noise — Google claims that Imagen Video is a step toward a system with a “high degree of controllability” and world knowledge, including the ability to generate footage in a range of artistic styles.
As my colleague Devin Coldewey noted in his piece about Make-A-Video, text-to-video systems aren’t new. Earlier this year, a group of researchers from Tsinghua University and the Beijing Academy of Artificial Intelligence released CogVideo, which can translate text into reasonably-high-fidelity short clips. But Imagen Video appears to be a significant leap over the previous state-of-the-art, showing an aptitude for animating captions that existing systems would have trouble understanding.
“It’s definitely an improvement,” Matthew Guzdial, an assistant professor at the University of Alberta studying AI and machine learning, told TechCrunch via email. “As you can see from the video examples, even though the comms team is selecting the best outputs there’s still weird blurriness and artificing. So this definitely is not going to be used directly in animation or TV anytime soon. But it, or something like it, could definitely be embedded in tools to help speed some things up.”
Image Credits: Google
Image Credits: Google
Imagen Video builds on Google’s Imagen, an image-generating system comparable to OpenAI’s DALL-E 2 and Stable Diffusion. Imagen is what’s known as a “diffusion” model, generating new data (e.g., videos) by learning how to “destroy” and “recover” many existing samples of data. As it’s fed the existing samples, the model gets better at recovering the data it’d previously destroyed to create new works.
Image Credits: Google
As the Google research team behind Imagen Video explains in a paper, the system takes a text description and generates a 16-frame, three-frames-per-second video at 24-by-48-pixel resolution. Then, the system upscales and “predicts” additional frames, producing a final 128-frame, 24-frames-per-second video at 720p (1280×768).
Image Credits: Google
Image Credits: Google
Google says that Imagen Video was trained on 14 million video-text pairs and 60 million image-text pairs as well as the publicly available LAION-400M image-text data set, which enabled it to generalize to a range of aesthetics. In experiments, they found that Imagen Video could create videos in the style of Van Gogh paintings and watercolor. Perhaps more impressively, they claim that Imagen Video demonstrated an understanding of depth and three-dimensionality, allowing it to create videos like drone flythroughs that rotate around and capture objects from different angles without distorting them.
In a major improvement over the image-generating systems available today, Imagen Video can also render text properly. While both Stable Diffusion and DALL-E 2 struggle to translate prompts like “a logo for ‘Diffusion’” into readable type, Imagen Video renders it without issue — at least judging by the paper.
That’s not to suggest that Imagen Video is without limitations. As is the case with Make-A-Video, even the clips cherrypicked from Imagen Video are jittery and distorted in parts, as Guzdial alluded to, with objects that blend together in physically unnatural — and impossible — ways. (To improve upon this, the Imagen Video team plans to combine forces with the researchers behind Phenaki, another Google text-to-video system that can turn long, detailed prompts into two-plus-minute videos, albeit at very low quality.) The researchers also note that the data used to train the system contained problematic content, which could result in Imagen Video producing graphically violent or sexually explicit clips; Google says it won’t release the Imagen Video model or source code “until these concerns are mitigated.”
Still, with text-to-video tech progressing at a rapid clip, it might not be long before an open source model emerges — both supercharging creativity and presenting an intractable challenge where it concerns deepfakes and misinformation.
Google answers Meta’s video-generating AI with its own, dubbed Imagen Video by Kyle Wiggers originally published on TechCrunch
DUOS
