Releasing resources and garbage collection

Previous tutorial: Image slide show combined with video

 

The scripting related tutorials we've seen so far were discussing different image manipulations performed with the variety of available plug-ins. Many of those manipulations process the specified image itself without allocating any new image objects. However some routines simply need to allocate new image objects to provide it as a result. This is valid for image processing plug-ins, which return result of a different pixel format and/or size or which calculate value of destination pixels based on the neighbours around them, etc. Same is valid for image importing plug-ins, which always allocate new image objects to return the loaded image. And so the logical question comes - who takes care of releasing resources occupied by the image objects allocated while running scripts?

The Lua Scripting plug-in introduced in Computer Vision Sandbox version 1.2.0 did not have any options to control when to release resources like image objects. It does not mean it did not take care of those at all. What it did was both simple and brutal - invoke garbage collection cycle after every run of a script's Main function. So if we had a camera providing 30 frames a second and the Lua Scripting plug-in was put into its video processing graph, then the scripting plug-in would trigger GC cycle 30 times per second. Even if the script does some simple image processing, which does not allocate any new image objects! Indeed, quite rough.

The scripting plug-in provided with Computer Vision Sandbox version 1.2.1 is more polite however and provides an option to control how often to trigger garbage collection cycle. By default it is set to 1, which means the same as the previous version of the plug-in did - collect garbage after every Main function call. Setting it to 10, for example, means to collect garbage after every 10 calls to Main. And so on. So increasing the number allows to invoke GC cycle less frequently. Finally it is possible to set it to 0, which switches off any GC triggering by the plug-in and leaves it all under control of the Lua library itself.

This is not all what is provided by the new version of the Lua Scripting plug-in however. The scripting plug-in provided with Computer Vision Sandbox version 1.2.1 also implements scripting API revision 2, which brings some new functions available for scripts. The function of interest now is Release(), which is available for image and plug-in objects. The Release() function simply frees resources occupied by the object it is called for. For plug-ins it is not that critical, since good scripts will create instances of all required plug-ins once before Main is run. Plus plug-in objects usually are not as expensive (resource wise) as image objects. However for image objects it is very different - those may get allocated on every call to Main, which is on every video frame if the script is used as a step in video processing graph. Invoking garbage collection every time after running a script is an option, but looks more like a work around. A better solution would be to explicitly release objects, which are no longer needed.

Let's have a look some samples of leaking image objects and see how to fix those.

-- Memory leak #1
resizePlugin = Host.CreatePluginInstance( 'ResizeImage' )

function Main( )
    -- Get image to process
    image = Host.GetImage( )
    -- Resize image
    resizePlugin:SetProperty( 'width',  image:Width( )  / 4 )
    resizePlugin:SetProperty( 'height', image:Height( ) / 4 )
    resizedImage = resizePlugin:ProcessImage( image )
    -- Put resized image on top of the original
    image:PutImage( resizedImage, 0, 0 )
end

The first one is a classical leak - new image object was allocated by image resizing plug-in, however it was never released after that. Better release it at the end of the Main function.

-- Memory leak #2
rotatePlugin = Host.CreatePluginInstance( 'RotateImage' )
rotatePlugin:SetProperty( 'angle', 10 )

function Main( )
    -- Get image to process
    image = Host.GetImage( )
    -- Rotate the image
    rotateImage = rotatePlugin:ProcessImage( image )
    -- Set image back to host
    Host.SetImage( rotateImage )
end

The second example demonstrates leak of the same type. The part which may potentially obfuscate it is the call to Host.SetImage(), which sets the rotated image back to host for further processing and displaying. However the host application will not take ownership of the image allocated in a script, it will simply copy its pixels' data. It is the script who needs to take care of the allocated image object.

-- Memory leak #3
imageImporterPlugin = Host.CreatePluginInstance( 'JpegImporter' )

function Main( )
    -- Load image
    loadedImage = imageImporterPlugin:ImportImage( [[c:\temp\my_image.jpg]] )
    -- Get image to process
    image = Host.GetImage( )
    -- Put resized image on top of the original
    image:PutImage( loadedImage, 0, 0 )
end

The last example is a bit artificial one. Instead of loading same static image again and again, it is much better to do it once before calling Main function. However lets pretend it is just a small sample - something more advanced will have some logic to decide which image file to load. Anyway, same leak as before - allocated image object is not released. The only difference here is that an image importing plug-in was used instead of an image processing one.

All we need to fix all these examples is to release the temporary image at the end of Main.

function Main( )
    ...
    -- Release the temporary image
    resizedImage:Release( )
end

Of course all of these leaks would be taken care by Garbage Collection cycle. However there is a trade of then - let GC run after every call to Main (too much) or let application consume resources and wait till GC gets in to free it (equally bad). My choice is to release resources explicitly whenever possible.

For a more real life example of taking care of image objects, it is recommended to have a look at the PicturesSlideshow.lua sample script, described in the previous tutorial. That tutorial did not make accent on releasing resources, however the script is well equipped for doing that.

 

Next tutorial: Sandbox variables