Adobe shows its own video AI Firefly Video for the first time
[18:08 Wed,11.September 2024 by Thomas Richter]
Adobe has been working on generative capabilities for video for a while – last year, Firefly for video was teased. Now, Adobe is showing its own AI-generated clips for the first time. These artificially created sequences aim to simplify daily editing tasks by conjuring up missing establishing shots or cutaway shots, or – for shots that are too short – generating extra frames for a transition.
According to Adobe, nature videos are particularly successful with its video model at the moment, such as landscapes, animals, or plants. This suggests that Firefly may not yet handle people in motion perfectly; such images are missing from the presented demo material. (However, practically all AI video generators currently struggle with human motion; only the recently presented Video-01 model from MiniMax has shown significant progress in this area.) Firefly for video, however, appears to create detailed views of human faces without issue.
For motion graphics or video animations, Firefly can generate moving elements like fire, smoke, dust, or water against black or blue backgrounds, which can then be overlaid in Premiere or After Effects via layer modes or a key with real footage. Firefly can also quickly visualize different ideas during project development, allowing them to be shared with clients or teams (even in different styles like line drawings or claymation).
Firefly Video GUI
Usage and Options
Firefly for video will generate clips from both text prompts (Text-to-Video) and reference images (Image-to-Video). You can also make fairly specific requests regarding the desired look and camera movements – in one example, a dolly zoom is requested alongside initial blurriness, and it is delivered (which is not surprising, as these clips are hand-picked).
The user interface shown in the demo clip contains additional hints about which parameters can be specified when generating video (though the final interface may still change before launch). These include aspect ratio, frame rate, shot size, camera angle, and camera movement. Also notable is a dropdown labeled "Model" – Adobe previously hinted at NAB that they wanted to integrate video models from other providers (such as OpenAI's Sora, Runway, etc.). This isn't explicitly mentioned in the current demo, but the corresponding menu option is still (or already) present.
Firefly Video Camera Controls
The maximum length for AI-generated sequences isn’t specified; all the shown clips are about 4 seconds long. According to Adobe, they were computed in less than 2 minutes. Since Adobe's Firefly video model is trained solely on licensed material, using the generated clips in commercial settings is also considered safe.
;)
;)
;)