Building simple motion detector

Previous tutorial: Two source image processing

 

The previous article gave a brief introduction to two source image processing filter plug-ins. Now it is time to explore them further and demonstrate one of their possible applications - building a simple motion detector.

One of the simplest motion detectors can be obtained by calculating difference between consecutive video frames: current and previous frames, for example. If the difference is lower than a certain threshold value, then it is treated as no motion detected. And if it is higher than the threshold, then we suppose something is moving. The idea of such motion detector is very basic and of course may even lie to us. The point is that it does not detect motion actually, but the fact something has changed between two video frames. Most of the time changes are caused by something moving. But not always. A change in illumination will also cause considerable difference between video frames and so will be treated as motion by such detector. However, is it really a big problem for us? Suppose we have a video surveillance camera, which needs to raise alarm and/or start saving video when something (like motion) is detected. Sudden change in illumination may indicate someone switched lights actually. And this already may need to be treated with caution, since presence of that someone may not be welcome.

To find difference between two images the Diff Images plug-in can be used, which calculates absolute difference between pixels of the two images. Bigger changes between two video frames result in higher number of pixels' values being different.

Getting an image indicating areas of difference is not enough however. What we also need is some single value indicating amount of changes, which could be then compared against some alarm threshold. To get such value we could simply count number of pixels, which have difference higher than certain limit. And for this purpose we have another plug-in, which does all the job for us - Diff Images Thresholded. This plug-in does more than Diff Images does - in a single pass it calculates difference between pixels, thresholds it to keep only significant changes, counts number of different pixels and colors the output image to show areas of difference. This one single plug-in implements most of what is required for our simple motion detector.

To demonstrate Diff Images Thresholded plug-in in action, below is a Lua script implementing complete motion detector based on the described above idea. When motion is detected, it highlights motion areas on the input image and surrounds it with a red rectangle. Also percent of changed pixels is displayed for debug purpose.

-- Create instances of plug-ins to use
diffImages   = Host.CreatePluginInstance( 'DiffImagesThresholded' )
addImages    = Host.CreatePluginInstance( 'AddImages' )
imageDrawing = Host.CreatePluginInstance( 'ImageDrawing' )

-- Since we deal with RGB images, set threshold to 60 for the sum
-- of RGB differences
diffImages:SetProperty( 'threshold', 60 )
-- Highlight motion area with red color
diffImages:SetProperty( 'hiColor', 'FF0000' )

-- Amount of diff image to add to source image for motion highlighting
addImages:SetProperty( 'factor', 0.3 )

-- Motion alarm threshold
motionThreshold = 0.1

function Main( )
    image = Host.GetImage( )

    if oldImage ~= nil then
        -- Calculate difference between current and the previous frames
        diff = diffImages:ProcessImage( image, oldImage )

        -- Set previous frame to the current one
        oldImage:Release( )
        oldImage = image:Clone( )

        -- Get the difference amount
        diffPixels  = diffImages:GetProperty( 'diffPixels' )
        diffPercent = diffPixels * 100 / ( image:Width( ) * image:Height( ) )

        -- Output the difference value
        imageDrawing:CallFunction( 'DrawText', image, tostring( diffPercent ),
                                   { 1, 1 }, 'FFFFFF', '00000000' )

        -- Check if alarm has to be raised
        if diffPercent > motionThreshold then
            imageDrawing:CallFunction( 'DrawRectangle', image,
                { 0, 0 }, { image:Width( ) - 1, image:Height( ) - 1 }, 'FF0000' )

            -- Highlight motion areas
            addImages:ProcessImageInPlace( image, diff )
        end

        diff:Release( )
    else
        oldImage = image:Clone( )
    end
end

The above demonstrated motion detector works really good taking into account its simplicity. One of its issues however, is that it does not highlight motion very accurately. If an object moves reasonably fast, then this detector will highlight as new position of the object, as its old position. And if the object is moving slowly, then the highlighted area looks more like a thick edge around the object. This motion detector does not have any notion of moving objects at all. The only thing it detects is difference. Which is fine in most cases.

One way to overcome the above issue would be to calculate difference not between two consecutive frames, but between current video frame and a background frame - a frame which contains only the background scene, not the moving objects. The problem is how to get the background frame or at least its reasonably good estimate. For the scenes like shown above, where camera is monitoring street view with constantly moving objects, there is one simple technique, which may help in modelling approximation of a background frame. Initially, when motion detector starts its job, it takes very first video frame as the background frame. Then on every new frame, the background frame is "moved towards" the current frame. Move Towards Images is the plug-in, which implements this idea - it changes pixels of one image in the direction to make difference with another image smaller (by certain value). If the plug-in is given two images A and B and it is applied again and again to move A to B, then eventually image A will become identical to image B. In the case of our motion detector, A is the background image and B is the current video frame. And so background is always updated in the direction of current frame. What is the result of it? If objects are constantly moving, then the background frame does not get too much time to get those objects. Yes, it will start changing by small fraction. But then objects move to different position and so background frame changes back to the surrounding scene. However, if a moving object stops movement, then after a while it will become part of the scene and so will appear on the background frame. This has a useful side effect actually. Suppose someone was walking on a street, left a bag, for example, and then disappeared. The bag, although is not moving, will be treated as an area of motion until background frame gets updated to contain it. Now someone comes, takes the bag and walks away. Again the missing bag will be treated as motion area until background frames updates appropriately.

Below script is very similar to the previous one, but implements the described above motion detector based on background frame modelling. Although the modelling technique is very simple, we can see that for the scene we are dealing with, motion highlighting has improved considerably and only shows where the moving objects are on the current video frame.

-- Create instances of plug-ins to use
diffImages   = Host.CreatePluginInstance( 'DiffImagesThresholded' )
addImages    = Host.CreatePluginInstance( 'AddImages' )
moveTowards  = Host.CreatePluginInstance( 'MoveTowardsImages' )
imageDrawing = Host.CreatePluginInstance( 'ImageDrawing' )

-- Since we deal with RGB images, set threshold to 60 for the sum
-- of RGB differences
diffImages:SetProperty( 'threshold', 60 )
-- Highlight motion area with red color
diffImages:SetProperty( 'hiColor', 'FF0000' )

-- Amount of diff image to add to source image for motion highlighting
addImages:SetProperty( 'factor', 0.3 )

-- Motion alarm threshold
motionThreshold = 0.1

-- Update background after every 5th frame
backgroundUpdateInterval = 5
counter = 0

function Main( )
    image = Host.GetImage( )

    if backgroundImage ~= nil then
        -- Calculate difference between current frame and the background
        diff = diffImages:ProcessImage( image, backgroundImage )

        -- Update background frame
        counter = ( counter + 1 ) % backgroundUpdateInterval
        if counter == 0 then
            moveTowards:ProcessImageInPlace( backgroundImage, image )
        end

        -- Get the difference amount
        diffPixels  = diffImages:GetProperty( 'diffPixels' )
        diffPercent = diffPixels * 100 / ( image:Width( ) * image:Height( ) )

        -- Output the difference value
        imageDrawing:CallFunction( 'DrawText', image, tostring( diffPercent ),
                                   { 1, 1 }, 'FFFFFF', '00000000' )

        -- Check if alarm has to be raised
        if diffPercent > motionThreshold then
            imageDrawing:CallFunction( 'DrawRectangle', image,
                { 0, 0 }, { image:Width( ) - 1, image:Height( ) - 1 }, 'FF0000' )

            -- Highlight motion areas
            addImages:ProcessImageInPlace( image, diff )
        end

        diff:Release( )
    else
        backgroundImage = image:Clone( )
    end
end

As the picture above demonstrates the result looks much better now. Of course it will not be always like this. There are many factors like objects speed, color, etc., which affect the result. Another important thing to mention is that if motion detection algorithm starts its work when there are already objects on the scene, they will be taken as background. And so when they move away, the motion detector will still detect motion for a while. But it should settle after a while and manage to update the background frame to contain only the scene. Well, for the environments like shown above. If you sit in front of your web camera waving your hands, this algorithm will not achieve any better result.

Below is an example of background frame the algorithm managed to get at the point of detecting motion shown on the above picture.

Finally, it is time to put something more into the script than just motion highlighting. In one of the previous tutorials it was shown how to use Video File Writer plug-in to do video saving. That tutorial demonstrated how to write video continuously regardless if there is any motion or not, which might be useful in certain cases to organize video archive, etc. However, with the motion detector we have, it is possible to write video only when there is motion detected (when changes happen). Below is a sample script which does this.

local os = require "os"

-- Create instances of plug-ins to use
diffImages   = Host.CreatePluginInstance( 'DiffImagesThresholded' )
addImages    = Host.CreatePluginInstance( 'AddImages' )
moveTowards  = Host.CreatePluginInstance( 'MoveTowardsImages' )
imageDrawing = Host.CreatePluginInstance( 'ImageDrawing' )
videoWriter  = Host.CreatePluginInstance( 'VideoFileWriter' )

-- Since we deal with RGB images, set threshold to 60 for the sum
-- of RGB differences
diffImages:SetProperty( 'threshold', 60 )
-- Highlight motion area with red color
diffImages:SetProperty( 'hiColor', 'FF0000' )

-- Amount of diff image to add to source image for motion highlighting
addImages:SetProperty( 'factor', 0.3 )

-- Motion alarm threshold
motionThreshold = 0.1

-- Update background after every 5th frame.
backgroundUpdateInterval = 5
counter = 0

-- Configure video file writer
videoWriter:SetProperty( 'folder', 'C:\\Temp\\camera' )
videoWriter:SetProperty( 'bitRate', 2000  )
videoWriter:SetProperty( 'syncPresentationTime', true )

-- Number of seconds to keep writing video after motion was detected
extraSaveTime = 5
-- Motion detection time (system clock, seconds)
motionDetectionTime = 0

videoFileStarted = false

function Main( )
    image = Host.GetImage( )

    if backgroundImage ~= nil then
        -- Calculate difference between current frame and the background
        diff = diffImages:ProcessImage( image, backgroundImage )

        -- Update background frame
        counter = ( counter + 1 ) % backgroundUpdateInterval
        if counter == 0 then
            moveTowards:ProcessImageInPlace( backgroundImage, image )
        end

        -- Get the difference amount
        diffPixels  = diffImages:GetProperty( 'diffPixels' )
        diffPercent = diffPixels * 100 / ( image:Width( ) * image:Height( ) )

        -- Output the difference value
        imageDrawing:CallFunction( 'DrawText', image, tostring( diffPercent ),
                                   { 1, 1 }, 'FFFFFF', '00000000' )

        -- Check if alarm has to be raised
        if diffPercent > motionThreshold then
            imageDrawing:CallFunction( 'DrawRectangle', image,
                { 0, 0 }, { image:Width( ) - 1, image:Height( ) - 1 }, 'FF0000' )

            -- Highlight motion areas
            addImages:ProcessImageInPlace( image, diff )
            -- Update time of detected motion
            motionDetectionTime = os.clock( )
        end

        repeater1:ProcessImage( diff )
        repeater2:ProcessImage( backgroundImage )
        diff:Release( )
    else
        backgroundImage = image:Clone( )
    end

    -- Check if we need to write current frame to a video file
    if os.clock( ) - motionDetectionTime <= extraSaveTime then
        if not videoFileStarted then
            -- Generate new file name
            videoWriter:SetProperty( 'fileName', os.date( '%Y-%m-%d %H-%M-%S' ) )

            videoFileStarted = true
        end

        -- Save current frame
        videoWriter:ProcessImage( image )
    else
        if videoFileStarted then
            -- Close video file
            videoWriter:Reset( )
            videoFileStarted = false
        end
    end
end

Well, that is it about motion detection for now. At its current stage it requires a bit of scripting to get it working. This gives certain level of flexibility however. In future versions of Computer Vision Sandbox there is a plan to introduce new plug-in types, which are aimed to perform different detection/recognition tasks. Those will encapsulate all the described steps required to implement motion detector and so will require close to zero coding, if at all.

 

Next tutorial: Plug-ins' functions

Copyright © 2015-2019 Andrew Kirillov