Capturing screen and windows
Previous tutorial: Creating time lapse video
The 1.2.2 version of Computer Vision Sandbox comes with a new Screen Capture plug-in, which is a video source allowing to capture content of system screens or applications' windows. This can be used for a number of different applications, like overlaying video with screen content or analysing content of certain windows to find objects of interest, for example. This tutorial will show how to get drawings from Paint application and put them on top of video coming from some video camera.
Note: To follow this tutorial it is required to have two video sources: one for whatever camera available and the other is Screen Capture video source, which is configured to capture window content having "Paint" in its title. Then both video sources are put into a sandbox, which will run some Lua scripts provided below.
To put drawings from Paint onto a video feed, we'll assume that all drawings are done on a black background. This will allow us to extract all the color objects easily and overlay them with another picture we'd like. However, to extract those drawings we need to find coordinates of the drawing area first. We could try finding those coordinates manually and hard code them, but this does not look like a robust solution - a change in Paint's window layout or its size would ruin all the effort. So we'll better automate it if we can. The first plug-in to use is theColor Filter image processing filter, which is used to turn all non-black colors into white. In other words, all black areas in original image will remain black, while anything else is set to white.
It is clear that we need to find coordinates of the biggest black area - the one which contains all the drawings we have. We can employ some blobs processing plug-ins for this, but there is one more step we need to do to prepare the image. Since blobs processing routines treat black areas as background and everything else as blobs/object, we may need to use the Invert plug-in to satisfy this requirement.
Now the object of our interest is the biggest white one. Its coordinates can be easily obtained using the Find Biggest Blob, which then allow us extracting the drawings from the original image.
Putting this all together produces the Lua script below, which is applied to the Screen Capture video source capturing content of Paint application. Once all image processing steps are done, it sets the extracted image as a host variable, which will be then picked by another script doing the overlay part.
-- Color Filter is used to turn all non-black colors into white colorFilter = Host.CreatePluginInstance( 'ColorFilter' ) colorFilter:SetProperty( 'redRange', { 0, 0 } ) colorFilter:SetProperty( 'greenRange', { 0, 0 } ) colorFilter:SetProperty( 'blueRange', { 0, 0 } ) colorFilter:SetProperty( 'fillOutside', true ) colorFilter:SetProperty( 'fillColor', 'FFFFFF' ) -- Instance of Invert plug-in to invert an image invert = Host.CreatePluginInstance( 'Invert' ) -- Find Biggest Blob plug-in is used to find on object with bigest area findBiggestBlob = Host.CreatePluginInstance( 'FindBiggestBlob' ) findBiggestBlob:SetProperty( 'sizeCriteria', 1 ) function Main( ) -- Get image coming from screen capture video source image = Host.GetImage( ) -- Apply color filter and locate the drawing background tempImage = colorFilter:ProcessImage( image ) invert:ProcessImageInPlace( tempImage ) findBiggestBlob:ProcessImage( tempImage ) if findBiggestBlob:GetProperty( 'area' ) ~= 0 then topLeft = findBiggestBlob:GetProperty( 'topLeft' ) bottomRight = findBiggestBlob:GetProperty( 'bottomRight' ) w = bottomRight[1] - topLeft[1] h = bottomRight[2] - topLeft[2] -- Get sub-image of the drawing area only and store it as a host variable subImage = image:GetSubImage( topLeft[1], topLeft[2], w, h ) Host.SetVariable( 'paintImage', subImage ) subImage:Release( ) end tempImage:Release( ) end
The next step is to put the extracted drawings on top of the images coming from some video camera. There are different ways of doing this, like using Merge Images two source image processing filter. To use it though, the source image must be pre-processed first, so the areas where drawing will be put are cleared first. For this purpose we need to prepare a mask image, which is white in all areas we want to keep in the source image and black in all other areas we want to clear. To get the mask image, we first apply Grayscale filter to the drawings image, then Invert it and finally Threshold it.
Now when the mask is ready, we use it to clear drawing areas in the source image using the Mask Image plug-in and then we combine the source with drawings using the Merge Images plug-in.
The script below puts all the words above into action. Plus, it does some extra infrastructure work like resizing drawings to match the size of images provided by camera.
-- Get instances of plug-in we need and configure them resizeImage = Host.CreatePluginInstance( 'ResizeImage' ) grayscaleFilter = Host.CreatePluginInstance( 'Grayscale' ) invert = Host.CreatePluginInstance( 'Invert' ) thresholdFilter = Host.CreatePluginInstance( 'Threshold' ) maskImage = Host.CreatePluginInstance( 'MaskImage' ) mergeImages = Host.CreatePluginInstance( 'MergeImages' ) thresholdFilter:SetProperty( 'threshold', 255 ) resizeImage:SetProperty( 'interpolation', 1 ) function Main( ) -- Get image coming from video camera image = Host.GetImage( ) -- Get image set by script handling screen capture paint = Host.GetVariable( 'paintImage' ) if paint then -- Resize paint image to the same size as video frame resizeImage:SetProperty( 'width', image:Width ( ) ) resizeImage:SetProperty( 'height', image:Height( ) ) resizedPaint = resizeImage:ProcessImage( paint ) -- Turn paint image into a mask image mask = grayscaleFilter:ProcessImage( resizedPaint ) invert:ProcessImageInPlace( mask ) thresholdFilter:ProcessImageInPlace( mask ) -- Apply mask to the camera image to clear areas of drawings maskImage:ProcessImageInPlace( image, mask ) -- Merge the camera image with the drawings mergeImages:ProcessImageInPlace( image, resizedPaint ) maskImage:Release( ) resizedPaint:Release( ) paint:Release( ) end end
From the image above it can be seen that the achieved result has some artefacts around edges of the painted objects. This is caused by the fact that objects' edges are blurred by image resizing and Paint's anti-aliasing. This blurred edges lead to removing more than needed from camera image when doing masking. At the same time these edges are not bright enough, so, when merged back into the source image, they look like dark edges around drawings.
Below is a bit different version of overlaying drawings with a video source. It has improvements around edges of the pained objects, but has some issues with colors of the objects - they look a bit different from the original drawings images.
-- Get instances of plug-in we need and configure them resizeImage = Host.CreatePluginInstance( 'ResizeImage' ) subtractImages = Host.CreatePluginInstance( 'SubtractImages' ) addImages = Host.CreatePluginInstance( 'AddImages' ) saturate = Host.CreatePluginInstance( 'Saturate' ) levelsLiner = Host.CreatePluginInstance( 'LevelsLinear' ) levelsLiner:SetProperty( 'redIn', { 0, 210 } ) levelsLiner:SetProperty( 'greenIn', { 0, 210 } ) levelsLiner:SetProperty( 'blueIn', { 0, 210 } ) saturate:SetProperty( 'saturateBy', -100 ) resizeImage:SetProperty( 'interpolation', 1 ) function Main( ) -- Get image coming from video camera image = Host.GetImage( ) -- Get image set by script handling screen capture paint = Host.GetVariable( 'paintImage' ) if paint then -- Resize paint image to the same size as video frame resizeImage:SetProperty( 'width', image:Width ( ) ) resizeImage:SetProperty( 'height', image:Height( ) ) resizedPaint = resizeImage:ProcessImage( paint ) -- Turn painting into grayscale image and increase its brighness a bit grayPaint = saturate:ProcessImage( resizedPaint ) levelsLiner:ProcessImageInPlace( grayPaint ) -- Clear source image by subtracting grayscale version of drawings subtractImages:ProcessImageInPlace( image, grayPaint ) -- Add drawings into the source image addImages:ProcessImageInPlace( image, resizedPaint ) grayPaint:Release( ) resizedPaint:Release( ) paint:Release( ) end end
Well, we managed to achieve the main part the tutorial was aimed for - to capture content of some window, extract something which represents interest for us and then use it somehow - combine with a video source. There is still some space for improvements in regards to merging blurred objects with images, but for now we can leave it as is.
The techniques demonstrated in this tutorial can be used for many other application as well. For example, it might be possible to monitor/track status of some applications by analysing changes in the content of their windows. Or, if nothing else, the scripts above can be used together with the virtual video camera and then some drawings can be done while having a conversion over Skype.
Next tutorial: Detection of square binary glyphs