Capturing screen and windows

Previous tutorial: Creating time lapse video

 

The 1.2.2 version of Computer Vision Sandbox comes with a new Screen Capture plug-in, which is a video source allowing to capture content of system screens or applications' windows. This can be used for a number of different applications, like overlaying video with screen content or analysing content of certain windows to find objects of interest, for example. This tutorial will show how to get drawings from Paint application and put them on top of video coming from some video camera.

Note: To follow this tutorial it is required to have two video sources: one for whatever camera available and the other is Screen Capture video source, which is configured to capture window content having "Paint" in its title. Then both video sources are put into a sandbox, which will run some Lua scripts provided below.

To put drawings from Paint onto a video feed, we'll assume that all drawings are done on a black background. This will allow us to extract all the color objects easily and overlay them with another picture we'd like. However, to extract those drawings we need to find coordinates of the drawing area first. We could try finding those coordinates manually and hard code them, but this does not look like a robust solution - a change in Paint's window layout or its size would ruin all the effort. So we'll better automate it if we can. The first plug-in to use is theColor Filter image processing filter, which is used to turn all non-black colors into white. In other words, all black areas in original image will remain black, while anything else is set to white.

It is clear that we need to find coordinates of the biggest black area - the one which contains all the drawings we have. We can employ some blobs processing plug-ins for this, but there is one more step we need to do to prepare the image. Since blobs processing routines treat black areas as background and everything else as blobs/object, we may need to use the Invert plug-in to satisfy this requirement.

Now the object of our interest is the biggest white one. Its coordinates can be easily obtained using the Find Biggest Blob, which then allow us extracting the drawings from the original image.

Putting this all together produces the Lua script below, which is applied to the Screen Capture video source capturing content of Paint application. Once all image processing steps are done, it sets the extracted image as a host variable, which will be then picked by another script doing the overlay part.

-- Color Filter is used to turn all non-black colors into white
colorFilter = Host.CreatePluginInstance( 'ColorFilter' )
colorFilter:SetProperty( 'redRange',   { 0, 0 } )
colorFilter:SetProperty( 'greenRange', { 0, 0 } )
colorFilter:SetProperty( 'blueRange',  { 0, 0 } )
colorFilter:SetProperty( 'fillOutside', true )
colorFilter:SetProperty( 'fillColor', 'FFFFFF' )

-- Instance of Invert plug-in to invert an image
invert = Host.CreatePluginInstance( 'Invert' )

-- Find Biggest Blob plug-in is used to find on object with bigest area
findBiggestBlob = Host.CreatePluginInstance( 'FindBiggestBlob' )
findBiggestBlob:SetProperty( 'sizeCriteria', 1 )

function Main( )
    -- Get image coming from screen capture video source
    image = Host.GetImage( )

    -- Apply color filter and locate the drawing background
    tempImage = colorFilter:ProcessImage( image )
    invert:ProcessImageInPlace( tempImage )
    findBiggestBlob:ProcessImage( tempImage )

    if findBiggestBlob:GetProperty( 'area' ) ~= 0 then
        topLeft     = findBiggestBlob:GetProperty( 'topLeft' )
        bottomRight = findBiggestBlob:GetProperty( 'bottomRight' )
        w = bottomRight[1] - topLeft[1]
        h = bottomRight[2] - topLeft[2]

        -- Get sub-image of the drawing area only and store it as a host variable
        subImage = image:GetSubImage( topLeft[1], topLeft[2], w, h )
        Host.SetVariable( 'paintImage', subImage )

        subImage:Release( )
    end

    tempImage:Release( )
end

The next step is to put the extracted drawings on top of the images coming from some video camera. There are different ways of doing this, like using Merge Images two source image processing filter. To use it though, the source image must be pre-processed first, so the areas where drawing will be put are cleared first. For this purpose we need to prepare a mask image, which is white in all areas we want to keep in the source image and black in all other areas we want to clear. To get the mask image, we first apply Grayscale filter to the drawings image, then Invert it and finally Threshold it.

Now when the mask is ready, we use it to clear drawing areas in the source image using the Mask Image plug-in and then we combine the source with drawings using the Merge Images plug-in.

The script below puts all the words above into action. Plus, it does some extra infrastructure work like resizing drawings to match the size of images provided by camera.

-- Get instances of plug-in we need and configure them
resizeImage     = Host.CreatePluginInstance( 'ResizeImage' )
grayscaleFilter = Host.CreatePluginInstance( 'Grayscale' )
invert          = Host.CreatePluginInstance( 'Invert' )
thresholdFilter = Host.CreatePluginInstance( 'Threshold' )
maskImage       = Host.CreatePluginInstance( 'MaskImage' )
mergeImages     = Host.CreatePluginInstance( 'MergeImages' )

thresholdFilter:SetProperty( 'threshold', 255 )
resizeImage:SetProperty( 'interpolation', 1 )

function Main( )
    -- Get image coming from video camera
    image = Host.GetImage( )
    -- Get image set by script handling screen capture
    paint = Host.GetVariable( 'paintImage' )
    
    if paint then
        -- Resize paint image to the same size as video frame
        resizeImage:SetProperty( 'width',  image:Width ( ) )
        resizeImage:SetProperty( 'height', image:Height( ) )
        resizedPaint = resizeImage:ProcessImage( paint )

        -- Turn paint image into a mask image
        mask = grayscaleFilter:ProcessImage( resizedPaint )
        invert:ProcessImageInPlace( mask )
        thresholdFilter:ProcessImageInPlace( mask )

        -- Apply mask to the camera image to clear areas of drawings
        maskImage:ProcessImageInPlace( image, mask )
        -- Merge the camera image with the drawings
        mergeImages:ProcessImageInPlace( image, resizedPaint )

        maskImage:Release( )
        resizedPaint:Release( )
        paint:Release( )
    end
end

From the image above it can be seen that the achieved result has some artefacts around edges of the painted objects. This is caused by the fact that objects' edges are blurred by image resizing and Paint's anti-aliasing. This blurred edges lead to removing more than needed from camera image when doing masking. At the same time these edges are not bright enough, so, when merged back into the source image, they look like dark edges around drawings.

Below is a bit different version of overlaying drawings with a video source. It has improvements around edges of the pained objects, but has some issues with colors of the objects - they look a bit different from the original drawings images.

-- Get instances of plug-in we need and configure them
resizeImage    = Host.CreatePluginInstance( 'ResizeImage' )
subtractImages = Host.CreatePluginInstance( 'SubtractImages' )
addImages      = Host.CreatePluginInstance( 'AddImages' )
saturate       = Host.CreatePluginInstance( 'Saturate' )
levelsLiner    = Host.CreatePluginInstance( 'LevelsLinear' )

levelsLiner:SetProperty( 'redIn',   { 0, 210 } )
levelsLiner:SetProperty( 'greenIn', { 0, 210 } )
levelsLiner:SetProperty( 'blueIn',  { 0, 210 } )

saturate:SetProperty( 'saturateBy', -100 )
resizeImage:SetProperty( 'interpolation', 1 )

function Main( )
    -- Get image coming from video camera
    image = Host.GetImage( )
    -- Get image set by script handling screen capture
    paint = Host.GetVariable( 'paintImage' )
    
    if paint then
        -- Resize paint image to the same size as video frame
        resizeImage:SetProperty( 'width',  image:Width ( ) )
        resizeImage:SetProperty( 'height', image:Height( ) )
        resizedPaint = resizeImage:ProcessImage( paint )

        -- Turn painting into grayscale image and increase its brighness a bit
        grayPaint = saturate:ProcessImage( resizedPaint )
        levelsLiner:ProcessImageInPlace( grayPaint )

        -- Clear source image by subtracting grayscale version of drawings
        subtractImages:ProcessImageInPlace( image, grayPaint )
        -- Add drawings into the source image
        addImages:ProcessImageInPlace( image, resizedPaint )

        grayPaint:Release( )
        resizedPaint:Release( )
        paint:Release( )
    end
end

Well, we managed to achieve the main part the tutorial was aimed for - to capture content of some window, extract something which represents interest for us and then use it somehow - combine with a video source. There is still some space for improvements in regards to merging blurred objects with images, but for now we can leave it as is.

The techniques demonstrated in this tutorial can be used for many other application as well. For example, it might be possible to monitor/track status of some applications by analysing changes in the content of their windows. Or, if nothing else, the scripts above can be used together with the virtual video camera and then some drawings can be done while having a conversion over Skype.

 

Next tutorial: Detection of square binary glyphs

Copyright © 2015-2019 Andrew Kirillov