Two source image processing

Previous tutorial: Sandbox variables

 

So far we've seen how to use image processing filter plug-ins from Lua scripting for processing images to achieve different results. Those allow processing a single image and produce a new image representing result of the processing or change the source image itself (if the plug-in allows that). However, there are image processing filters, which require more than one image as input. For example, adding/subtracting images, finding intersection (minimum) or merging images (maximum), etc. These types of routines can not be implemented within the image processing filter's interface we've used so far.

The Lua scripting API revision 3 introduces support of few more plug-in types. One of them is two source image processing filters. Plug-ins implementing this interface take two images as input. Similar to the single source image processing filters, the two source filters can either produce a new image as a result or change the first image if possible.

The two source image processing filter plug-ins are not seen in the user interface of sandbox wizard and so can not be added as a step into video processing graph. So far the only way to use them is from scripting. A question may arise: "How to find these plug-ins, if they are not visible in sandbox wizard?" The 1.2.2 version of Computer Vision Sandbox extends its plug-ins' help system and allows browsing the list of modules loaded by the application and the list of plug-ins available in every module. Exploring those provides information about plug-ins, their types, properties, what they do, etc.

Now, lets have a look at some samples. Will not go through every available two source image processing filter plug-in here, but see the most common ones and check how to use them. Below is a source image, which will be used to demonstrate some of the plug-ins.

Suppose we applied some image processing filter plug-ins to the above image and got two grayscale images showing areas of blue and red objects. It is not the topic of this tutorial how we got to these two images. However if someone wants to try it, then extracting nRGB channels, thresholding them and finally some blobs filtering is the direction to dig into. The rest we'll keep for further tutorials.

One of the things we may want to do is to get the red and blue detected objects on one image. For this purpose we can use Merge Images plug-in, which implements minimum operator - for every two pixels of the source images it sets pixels in result image to the minimum value of those. 

-- create plug-in instance
mergeImages = Host.CreatePluginInstance( 'MergeImages' )

-- merge two image and get result in a new image
mergedImage = mergeImages:ProcessImage( redObjects, blueObjects )

-- if we don't need the first image afterwards, we may have result
-- to be put in it to save memory
mergeImages:ProcessImageInPlace( redObjects, blueObjects )

Another useful plug-in is Intersect Images, which implements maximum operator. We can use that to extract color objects from the original image. For this we'll need to convert the grayscale mask we have to RGB image using Grayscale to RGB plug-in. Alternatively we could use Mask Image plug-in to apply a mask to the source image. But we'll need the RGB color mask for some other filters as well anyway. 

-- create plug-in instance
intersectImages = Host.CreatePluginInstance( 'IntersectImages' )

-- intersect original image with RGB mask image
colorObjects = intersectImages:ProcessImage( originalImage, maskImage )

Some more plug-ins to mention are Add Images and Subtract Images, which add and subtract pixels' values as the name of plug-ins suggest. These plug-ins can be used for different purposes, but what we can do here is to make our detected objects a bit highlighted or maybe darker. 

-- create plug-in instance
addImages = Host.CreatePluginInstance( 'AddImages' )

-- highlight detected objects by adding 20% of the mask image
addImages:SetProperty( 'factor', 0.2 )
highlightedObjects = addImages:ProcessImage( originalImage, maskImage )


-- create plug-in instance
subtractImages = Host.CreatePluginInstance( 'SubtractImages' )

-- make detected objects darker by subtracting 30% of the mask image
subtractImages:SetProperty( 'factor', 0.3 )
darkerObjects = subtractImages:ProcessImage( originalImage, maskImage )

Combining two source image processing filters with single source filters can bring some other interesting results. Below picture demonstrates changing colors of the detected objects. First a simple Rotate RGB Channels plug-in was applied to the image containing our detected object only. Then the mask used to extract those objects was inverted by the Invert plug-in and used with Intersect Images plug-in again on the original image, to get everything except object. Finally the re-colored objects were merged back into the scene.

Well, we'll leave the topic for now and let everyone to explore the new two source image processing filters themselves. Of course more of these will appear in other tutorials, like the Embed Quadrilateral plug-in shown in the previous one - a very useful plug-in for embedding one image into arbitrary quadrilateral of another image.

 

Next tutorial: Building simple motion detector