Optical flow

Motion vectors that caused a quick tracking shot to a target in the lower center of the image. Here the vectors come from the motion estimation, which are stored in MPEG data, and are therefore not a regular vector field for every pixel in the picture.

The optical flow of an image sequence is the vector field of the projected speed of visible points of the object space in the reference system of the imaging optics into the image plane .

Optical flow is an important representation of motion information in early stages of image processing . Like the segmentation according to colors and textures, it supports the assignment of points to objects. Application examples are the optical computer mouse , the image stabilization of cameras, the motion compensation in video compression and the visual navigation of robots and animals.

The (desired) result of a calculation method or even the method used is also referred to as the optical flow. Important methods are differential methods, which usually work in pixels, and methods that work in blocks.

methodology

The local optical flow is a considered pixel from patterns in the picture in a more or less large area estimated . Only the component of the flow vector that is parallel to the gradient can be determined from the local gradient . This fundamental problem is called the aperture problem .

Whether the vector of interest can be precisely determined therefore depends on whether there are gray value gradients in different directions within the considered area G. In addition, it is necessary to have a model of the basic course of the optical flow within the considered area; in the simplest case it is assumed that the optical flux can be regarded as constant within small areas. More complicated courses of the flow field (e.g. affine models) are possible and are used in powerful processes. An interest operator supplies those points whose flow vector can be determined particularly reliably. In some approaches, the flow is only calculated at these selected points ( feature point tracking ).

Further methods are least-squares matching and block correlation (minimized sum of the absolute differences , normalized cross-correlation). A special form of block-wise motion estimation is the phase correlation based on the Fourier transformation (inversion of the normalized cross-power density spectrum ).

Differential optical flow

The calculation of the optical flow using differential methods goes back to the method developed by Berthold Horn and Brian Schunck at MIT in 1981 .

It is assumed that the brightness is constant at corresponding points in the individual images in the image sequence. Then it follows from the derivation ${\ displaystyle E}$

{\ displaystyle {\ frac {dE} {dt}} = {\ frac {\ partial E} {\ partial t}} + {\ frac {\ partial E} {\ partial x}} {\ frac {dx} { dt}} + {\ frac {\ partial E} {\ partial y}} {\ frac {dy} {dt}}}

as a necessary condition the determining equation for the velocities:

{\ displaystyle {\ frac {\ partial E} {\ partial t}} + {\ frac {\ partial E} {\ partial x}} {\ frac {dx} {dt}} + {\ frac {\ partial E } {\ partial y}} {\ frac {dy} {dt}} = 0.}

(Compare continuity equation )

The solution of this equation is a poorly posed problem in the sense of Jacques Hadamard . Therefore, the solution also requires smoothness.

There are several methods of determining the optical flow, including:

Some well-known algorithms for calculating the optical flow are implemented in the OpenCV library.

Applications

Object tracking: we humans can easily recognize a moving object based on its movement. In this way we can also distinguish between different objects based on their different movement. The calculation of the optical flow is therefore often used as a support in order to first identify a certain object and then to track its movement within an image sequence. It is assumed that the gradient vectors of an object between two images are approximately the same size and change evenly but not abruptly within the image sequence.
Image segmentation (English image segmentation.): Just as with the object tracking, the idea is also used in an image segmentation. So z. B. identify the stationary background or different areas with similar movement patterns.
Mobile robots : mobile robots equipped with cameras have to navigate their surroundings, recognize obstacles and avoid them. Methods for calculating the optical flow are used for object tracking and image segmentation.
Video Compression : Some processes for video compression exploit the fact that successive images change continuously or in part, not at all. The movement is calculated and saved using an optical flow. Unmoving image areas can be omitted and are simply taken over from the last single image. This can reduce the size of the video.
Motion detection (engl motion capture.): The optical flow is also used in techniques for motion detection used. These include B. the recognition of gestures and facial expressions or the recording of complex movements of a whole person for realistic computer animation in cartoons.

Optical flow in nature

With insects

Bees and other insects with compound eyes use the optical flow to

to fly straight ahead: The rate of rotation is regulated so that the attractor and repeller of the optical flow are diametrically opposite one another.
Avoiding obstacles: The flight direction is changed to a direction with little optical flow.
and estimate distances: The greater the maximum of the optical flow at a given airspeed, the smaller the distance. This way, flies can land on the ceiling by turning into the supine position in good time.

Human physiology

Similar to insects, the optical flow helps us in pedestrian and road traffic : We perceive the movement of other road users out of the corner of our eye and unconsciously take them into account when we move around. Objects with divergent flow come closer.

swell

^ Berthold KP Horn, Brian G. Schunck: Determining optical flow. In: Artificial Intelligence . 17 (1-3), 1981, pp. 185-203. doi : 10.1016 / 0004-3702 (81) 90024-2 .

literature

Amar Mitiche, JK Aggarwal: Computer Vision Analysis of Image Motion by Variational Methods , Springer, Cham / Heidelberg / New York / Dordrecht / London 2014, ISBN 978-3-319-00711-3 (eBook)

Web links

centeye.com: What is "Optical Flow"? ( Memento of April 17, 2010 in the Internet Archive )
Optical Flow (English)

[1] Berthold KP Horn, Brian G. Schunck: Determining optical flow. In: Artificial Intelligence . 17 (1-3), 1981, pp. 185-203. doi : 10.1016 / 0004-3702 (81) 90024-2 .