[14:14 Tue,21.January 2020 by Thomas Richter] |
In many areas, new AI/Deep Learning algorithms are revolutionizing the way we work with video - one area of this is the intelligent interpolation of frame rates, i.e. the calculation of one (or even several) frames between two existing frames of a video. The problem is to interpolate movements of objects in a way that they are logically continued on the synthesized images. This can only work really well if the algorithm can not only identify objects (and parts of them) and interpolate their movement and shape changes, but also recognize in which spatial depth which object is located in order to correctly predict mutual occlusions.
This (and more) is done by the new DAIN (Depth-Aware Video Frame Interpolation) algorithm developed by a research team from the universities of Shanghai and California plus Google. It uses four different neural networks (one each for Optical Flow Estimation, Depth Estimation, Content Extraction and Kernel Estimation) for the different problems of correct interpolation and then synthesizes their results into a video. The result works amazingly well - as you can see from the embedded sample videos below - and demonstrates what is state-of-the-art in frame interpolation. This is not only useful for making animations more fluid afterwards - (old) 24p footage can also be interpolated to 60fps afterwards. Another application would be to record 8K e.g. as a single frame sequence and subsequently pump it up to a higher frame rate. In the following video a 16x smooth slow motion is calculated by DAIN: Also the limitation of  REDs new Komodo camera of 6K with 40 fps could be lifted by interpolated 60p. ;) The DAIN Workflow Someone has kindly put together code from DAIN in the form of a DAIN app for Windows. This way, every user can easily test how well the interpolation of the frame rate via DAIN works on the basis of his own video material - however, CUDA is required for the calculation.  DAIN App The app is still at an early stage and is not yet optimized (and therefore 1 GB in size). Depending on the GPU power, the calculations can take quite a long time. Post your experiences, we are curious! Even more examples: Harryhausen animations:  more infos at bei sites.google.comdeutsche Version dieser Seite: Von 30fps zu 480fps: Flüssige Bildraten und extreme Zeitlupen Dank KI |
 |
|
|