[17:49 Sat,5.October 2024 by Thomas Richter] |
Meta has introduced Movie Gen, a new AI model for video generation, which brings several interesting features. In addition to the ability to create high-quality videos with fitting soundtracks or sound effects via text prompts, personalized videos can also be generated using a portrait photo. However, the standout feature is the ability to precisely manipulate objects in videos via prompts.
In fact, Meta’s Movie Gen consists of a collection of state-of-the-art AI models that enable these features when combined. The videos generated by Movie Gen are up to 16 seconds long and available in various formats. With a resolution of 1080p and 16 frames per second, along with audio, Meta claims the video quality is on par with the best current video AI models. More details below. ;) Meta’s video AI Movie Gen Movie Gen "understands" the typical movement patterns of various objects, the interactions between them, and can also perform specific camera movements—all the hallmarks of current video AI models, as demonstrated impressively by OpenAI’s Sora, as well as leading Chinese models like  ByteDance’s Seaweed and Minimax Hailuo and Kuaishou’s Kling. The official demo (see above) makes it clear that the quality of Movie Gen is extremely good—human movements look very realistic, the image details are sharp, and there are very few visual errors—at least in the Meta-preselected demo clips. How good Movie Gen truly is will only be revealed once users can freely use it.
Fortunately, Meta—like it has done multiple times before—takes a much more open approach to sharing its research results compared to other providers of video AI, and has made the related paper public, although the models themselves are not. This allows other developers, provided they have the necessary training data and computing power, to soon achieve similarly good results based on Meta’s findings. What are the special features of Movie Gen? Precise Video Editing via PromptThe standout feature and a glimpse into the future of video editing is the function for precise video editing via prompt: objects as well as the entire background can be specifically modified via prompt, and the visual style of a video can also be adjusted—without the appearance of adjacent objects being affected or other image elements changing their appearance. ;) Meta’s Video AI Movie Gen with Editing This is similar to the Video-to-Video function of Runway Gen-3 Alpha, but in direct comparison, it is much more precise, as only the desired objects are changed while the rest—unlike in Runway—remains completely untouched.
In the following video, Movie Gen&s editing feature (aka Video-to-Video) is explicitly compared with Runway Gen-3 and performs significantly better—the background remains untouched, and the integration of new objects looks much more seamless.
Movie Gen handles complex VFX tasks like replacing backgrounds, swapping or removing characters, and adding particle effects, all via simple text prompts.
Here’s a video that Meta CEO Mark Zuckerberg posted on Instagram with the announcement that Movie Gen will be integrated into Instagram next year:
Movie Gen Audio: Soundtracks and SFXThe video-to-audio model responsible for sound is called Movie Gen Audio—it can generate appropriate sound, such as ambient noises, instrumental background music, and sound effects (foley), for a video along with an optional text prompt. So far, only https://www.slashcam.de/news/single/Pika-integriert Weitere Bilder zur Newsmeldung:
 more infos at bei ai.meta.comdeutsche Version dieser Seite: Neue Video-KI Movie Gen kommt mit Killerfeature |
;)
;)
;)
 |
|
|