Lucas-Kanade method

The Lucas-Kanade method for calculating the optical flow goes back to the two researchers Bruce D. Lucas and Takeo Kanade . They first proposed this method in 1981. The method is a popular procedure that is still widely used today. The additional condition that is needed to calculate the optical flow is the assumption of the equality of the flow in the local area of ​​the central pixel for which the flow is determined.

Mathematical basics

The Lucas-Kanade method is based on the basic equation of optical flow . The flow for two 3D image volumes (2D or nD cases are similar) is given by . In a small environment with its center in the voxel , the flow is seen as constant. This assumption generally applies if the time steps between the images are chosen to be small enough. , , , The partial derivatives of the picture denote -, -, direction and time. If the voxels are numbered with , a system of equations can be set up: ${\displaystyle (V_{x},V_{y},V_{z})}$ ${\displaystyle m\times m\times m}$ ${\displaystyle m>1}$ ${\displaystyle x,y,z}$ ${\displaystyle I_{x}}$ ${\displaystyle I_{y}}$ ${\displaystyle I_{z}}$ ${\displaystyle I_{t}}$ ${\displaystyle x}$ ${\displaystyle y}$ ${\displaystyle z}$ ${\displaystyle 1\dotsc n}$ ${\displaystyle n=m^{3}}$

${\displaystyle I_{x_{1}}V_{x}+I_{y_{1}}V_{y}+I_{z_{1}}V_{z}=-I_{t_{1}}}$

${\displaystyle I_{x_{2}}V_{x}+I_{y_{2}}V_{y}+I_{z_{2}}V_{z}=-I_{t_{2}}}$

${\displaystyle \vdots }$

${\displaystyle I_{x_{n}}V_{x}+I_{y_{n}}V_{y}+I_{z_{n}}V_{z}=-I_{t_{n}}}$

This gives us more than three equations for the three flow variables we are looking for. There is an over-determined system. The following applies:

${\displaystyle {\begin{bmatrix}I_{x_{1}}&I_{y_{1}}&I_{z_{1}}\\I_{x_{2}}&I_{y_{2}}&I_{z_{2}}\\\vdots &\vdots &\vdots \\I_{x_{n}}&I_{y_{n}}&I_{z_{n}}\end{bmatrix}}{\begin{bmatrix}V_{x}\\V_{y}\\V_{z}\end{bmatrix}}={\begin{bmatrix}-I_{t_{1}}\\-I_{t_{2}}\\\vdots \\-I_{t_{n}}\end{bmatrix}}}$

The overdetermined system can now be solved using the least squares method:

${\displaystyle A{\vec {v}}=-b}$

${\displaystyle A^{T}A{\vec {v}}=A^{T}(-b)}$

or

${\displaystyle {\vec {v}}=(A^{T}A)^{-1}A^{T}(-b)}$

or

${\displaystyle {\begin{bmatrix}V_{x}\\V_{y}\\V_{z}\end{bmatrix}}={\begin{bmatrix}\sum I_{x_{i}}^{2}&\sum I_{x_{i}}I_{y_{i}}&\sum I_{x_{i}}I_{z_{i}}\\\sum I_{x_{i}}I_{y_{i}}&\sum I_{y_{i}}^{2}&\sum I_{y_{i}}I_{z_{i}}\\\sum I_{x_{i}}I_{z_{i}}&\sum I_{y_{i}}I_{z_{i}}&\sum I_{z_{i}}^{2}\\\end{bmatrix}}^{-1}{\begin{bmatrix}-\sum I_{x_{i}}I_{t_{i}}\\-\sum I_{y_{i}}I_{t_{i}}\\-\sum I_{z_{i}}I_{t_{i}}\end{bmatrix}}}$

The sum runs from i = 1 to n.

The flow can thus be determined on the images by calculating the derivatives (= gradients). In order to give the central voxel more weight, a weighting formula W (i, j, k) , with is often used . Gaussian functions can be used for this. Other extensions of the Lucas-Kanade method use statistical methods to better deal with noise. ${\displaystyle i,j,k\in [1,m]}$

This method is also used in a hierarchical process in which the flow is first calculated on a coarser scale and then gradually refined on an increasingly finer scale.

properties

One of the characteristics of the Lucas-Kanade method is that (like other local methods of calculating optical flow) it does not provide a dense flow (i.e. sparse , not dense ). The flow information quickly disappears with the distance from the edges (edges or corners). The advantage of the method is the relative robustness against noise and smaller defects in the image.

literature

• Lucas BD and Kanade T 1981, An iterative image registration technique with an application to stereo vision. Proceedings of Imaging understanding workshop , pp 121--130 ( pdf ).
• Lucas BD 1984, Generalized Image Matching by the Method of Differences , Dissertation.

Web links

• Image stabilization for ImageJ based on the Lucas – Kanade method.
• KLT : Kanade – Lucas – Tomasi Feature Tracker
• Takeo Kanade