Video stabilization via AI: Better than anything before?

[19:46 Mon,12.April 2021 by Thomas Richter]

The subsequent stabilization of shaky video recordings (especially from smartphone recordings) is an important task for many users and its success depends entirely on the method used. Especially with mobile videos, algorithms are faced with special challenges such as - especially with low-light recordings - noisy images due to the small sensors of cell phones or which often also suffer from strong motion blur and shaky artifacts due to the rolling shutter effect.

Basically, there are two methods to choose from, but each has specific weaknesses. First, the video is stabilized by tracking, cropping and enlarging as much of the image as possible, with the result that a more or less large (depending on the strength of the image wobble) peripheral area of the image is eliminated.

Alternatively, full images are generated by sequential motion estimation, motion smoothing, and generation of stabilized new single frames, but the price is clearly noticeable (and annoying) distortion artifacts, especially at the edges.

The new method in comparison

A new method

FuSta - Hybrid Neural Fusion for Full-frame Video Stabilization by a team from the National University Taiwan and Google promises the best of both worlds: video stabilization of the entire image without artifacts. It does so by using a Convolutional Neural Network to leverage neighboring frames to synthesize image content missing due to wobble at the edges for stabilized full-frame video, remove blurry images, and minimize artifacts caused by fast-moving objects.
And indeed: in comparison with current stabilization methods as well as the Warp Stabilizer in Adobe Premiere Pro, the new method performs very well and produces significantly fewer artifacts despite uncropped video.

However, even this technique is not perfect: Image errors also appear here when the camera moves too quickly at the edge of the stabilized material, and areas often appear somewhat wobbly, which can also be due to the original material. But, as Károly Zsolnai-Fehér of "Two Minute Papers" likes to remark, "two more papers down the line", i.e. two research papers later, these problems will probably have disappeared.

Here is the Two Minute Papers clip about the new process:

If you want to try the video stabilization via FuSta yourself and have the necessary prior knowledge you can download the code

here.

more infos at bei www.youtube.com

deutsche Version dieser Seite: Neue Methode zur Videostabilisierung per KI: Besser als alle bisherigen?