Lua Scripting API

The Lua Scripting plug-in for the Computer Vision Sandbox allows running Lua scripts which can interact with the application and perform certain custom actions. The application exposes certain API which enables scripting plug-in to talk to it. That API is then wrapped by the plug-in and is exposed to scripts in the manner suitable for the programming language supported by the plug-in, Lua in this case. So in addition to the standard language capabilities the scripts get access to the extra API, which allows them to do something useful while running in the environment of the Computer Vision Sandbox application. This API is a topic of this page and so will be described in details below. Note: for the standard Lua programming language capabilities, its syntax, etc. please refer to the Lua manual.

At this point the Lua Scripting plug-in can be used only as a step in a video processing graph and so the API concentrates mostly on getting an image from the host application, processing it with different plug-ins and then delivering it back to the host, where it is processed by further steps of a graph or gets displayed. While Computer Vision Sandbox gets extended and starts using scripting engine for more, the API may get extended to provide more capabilities.

Lua script structure

Lua scripts executed by Lua Scripting plug-in consist of two main parts - global initialization part and a Main function. Initialization part is called once usually and is aimed to perform whatever initialization required for a script to run further. If it is required to create instances of plug-ins to be used later or perform other time consuming job, then initialization part is the right place to do it. The Main function is something, which is usually called repeatedly and performs the main job a particular script is designed for. For example, in the case if Lua script is used as a step in a video processing graph, then scripts initialization part is done immedciately after starting the video source the graph is attached to and the Main function is called for every new video frame going through the graph.

Host interface

The Host interface of the Lua scripting API allows to interact with a host application, Computer Vision Sandbox in this case. Below is the list of methods exposed by the interface:

  • string Host.Name( ) - Provides name of the host application running the scripting engine plug-in.
  • string Host.Version( ) - Provides version of the host application.
  • pluginObject Host.CreatePluginInstance( pluginName:string ) - Creates instance of the specified plug-in and returns its instance on success.
  • imageObject Host.GetImage( ) - Gets current image from the host to process, if it is available. In the case if script is running in a video processing graph, the method is supposed to be called from Main function only, since there are no images to process at the initialization phase.
  • Host.SetImage( image:imageObject ) - Sets image back to host for further processing in a graph or displaying. Same as GetImage() - to be called from Main.
  • Host.SetVariable( name:string, var:variant ) - Stores variable on the host side. The allowed variables' types are all of the supported property types (see below), i.e. Lua scalar types and arrays, as well as image objects. This and GetVariable() methods allow several running scripts to exchange variables between each other. Note: setting variable to Nil removes it from host. API 3.
  • variant Host.GetVariable( name:string ) - Gets variable stored on the host side. API 3.
  • pluginObject Host.GetVideoSource( ) - Gets video source object for which the script is running for. Allows to get access to video source's run time properties, if any. API 6.

-- Create instance of a plug-in
setHuePlugin = Host.CreatePluginInstance( 'Grayscale' )

-- Main function to be executed for every frame
function Main( )
    -- Get image to process
    image = Host.GetImage( )
    -- Apply plug-in
    grayImage = setHuePlugin:ProcessImage( image )
    -- Set image back to host
    Host.SetImage( grayImage )
end

Image class interface

Image objects are the ones provided by Host.GetImage() API and are those to be processed by different image and video processing plug-ins. Here is the list of methods exposed by the image class interface:

  • imageObject Image.Create( width:number, height:number, pixelFormat:string ) - Creates an image of the specified size and format.
  • imageObject:Release( ) - Release reference to the image. The object can not be used after that. API 2.
  • number imageObject:Width( ) - Returns width of the image object.
  • number imageObject:Height( ) - Returns height of the image object.
  • number imageObject:BitsPerPixel( ) - Returns number of bits used to encode single pixel of the image object.
  • string imageObject:PixelFormat( ) - Returns string name of the image's pixel format. Possible values are: "Gray8", "RGB24", "RGBA32", "Gray16", "RGB48", "RGBA64", "Binary1", "Indexed1", "Indexed2", "Indexed4", "Indexed8".
  • imageObject imageObject:Clone( ) - Creates a copy of the image by cloning it. API 2.
  • imageObject imageObject:GetSubImage( x:number, y:number, width:number, height:number ) - Returns sub-image of the image. The sub-image starts at the specified X/Y coordinates of the source image and has the specified size. The sub-image must be completely within the source image's boundaries. API 2.
  • imageObject:PutImage( imageToPut:imageObject, x:number, y:number ) - Puts the specified image into the source image at the specified location. The coordinates can be out of the source image's boundaries. The method will put only the region which overlaps with the source image. API 2.

Plug-in base class interface

Instances of plug-ins' objects are created using Host.CreatePluginInstance() API. Every plug-in type inherits from a base plug-in class, which provides common interface of all plug-ins available in the Computer Vision Sandbox.

  • pluginObject:Release( ) - Release reference to the plug-in. The object can not be used after that. API 2.
  • string pluginObject:Name( ) - Returns name of the plug-in as seen in user interface.
  • string pluginObject:ShortName( ) - Returns scripting name of the plug-in which is used to create an instance of the plug-in.
  • string pluginObject:Version( ) - Returns version of the plug-in.
  • string pluginObject:Type( ) - Returns plug-in's type. Possible values are: "ImageProcessingFilter", "VideoProcessing", "ImageImporter", "ImageExporter", etc (to be documented once officially released).
  • variant pluginObject:GetProperty( propertyName:string ) - Gets value of the the specified property. Type of the value depends on the property type.
  • pluginObject:SetProperty( propertyName:string, value:variant ) - Sets value of the specified plug-in's property. Type of the value must correspond to the property type.
  • variant pluginObject:GetProperty( propertyName:string, index:integer ) - Gets individual value of an array's type property. For plug-ins, which have properties of array types, this method allows to retrieve individual elements of the array instead of entire array. Indexing starts with 1. API 3.
  • pluginObject:SetProperty( propertyName:string, index:integer, value:variant ) - Sets individual element of an array type property. Type of the value must correspond to the array type of the property. Indexing starts with 1. API 3.
  • [variant] pluginObject:CallFunction( functionName:string [, ... variant args ] ) - Calls a function exposed by the plug-in. Returns nil, if the invoked function returns nothing. API 3.

Plug-in property types to Lua types mapping

Below is the table, which shows mapping between plug-ins' properties types (which is shown in plug-in's description when scrpting information is enabled) and Lua types:

Plug-in property type Lua type
Boolean boolean
String string
Integer number
Floating point number
Range table

The Range property type is represented as a table in Lua, which has two values: table[1] - range's minimum value; and table[2] - range's maxim value.
Color table or string

When getting a Color property, it is always returned as a table containing 4 RGBA values: table[1] - Red, table[2] - Green, table[3] - Blue and table[4] - Alpha. When setting a Color property it can be set either as table or string. If setting as table, it can have 3 or 4 values - alpha value is optional and defaults to 255. If setting as string, then the string format is either "RRGGBB" or "AARRGGBB", where color values are specified as HEX values.
Point table

The Point property type is represented as a table in Lua, which has two values: table[1] - point's X value; and table[2] - range's Y value. API 3.
Size table

The Size property type is represented as a table in Lua, which has two values: table[1] - size's width value; and table[2] - size's height value. API 3.
Array table, with indexes starting from 1. API 3.
2D Array table of tables, where all sub-tables must be of the same length. API 5.
-- Sample of setting properites of different types

-- Create instances of two plug-ins
contrastPlugin    = Host.CreatePluginInstance( 'ContrastCorrection' )
colorFilterPlugin = Host.CreatePluginInstance( 'ColorFilter' )

-- Numeric property
contrastPlugin:SetProperty( 'factor', 0.5 )

-- Boolean property
colorFilterPlugin:SetProperty( 'fillOutside', false )

-- Range property
colorFilterPlugin:SetProperty( 'rangeRed', { 0, 50 } )
-- or
range = {}
range[1] = 0
range[2] = 50
colorFilterPlugin:SetProperty( 'rangeGreen', range )
colorFilterPlugin:SetProperty( 'rangeBlue',  range )

-- Color property
colorFilterPlugin:SetProperty( 'fillColor', { 255, 174, 201 } )
-- or same as string
colorFilterPlugin:SetProperty( 'fillColor', "FFAEC9" )
-- Sample of getting array properties (API 3)

-- Create an instance of Image Statistics plug-in
statsPlugin = Host.CreatePluginInstance( 'ImageStatistics' )
-- Process image with it to get statistics values
statsPlugin:ProcessImage( image )

-- Get red histogram and check number of pixels having 0 and 255 value
-- (Lua arrays start with index 1)
redHist  = statsPlugin:GetProperty( 'redHistogram' )
count0   = redHist[1]
count255 = redHist[256]

-- Get same values individually without getting entire array
-- (Indexing starts with 1 to keep in sync with Lua arrays)
count0   = statsPlugin:GetProperty( 'redHistogram', 1 )
count255 = statsPlugin:GetProperty( 'redHistogram', 256 ) 
-- Sample of setting 2D array properties (API 5)

-- Create an instance of Convolution plug-in
convolutionPlugin = Host.CreatePluginInstance( 'Convolution' )
-- Set its kernel, which is a 2D array
kernel2d =
{
    { -1, 0, 1 },
    { -2, 0, 2 },
    { -1, 0, 1 }
}
convolutionPlugin:SetProperty( 'kernel', kernel2d )

Image processing filter plug-in interface

Image processing filter plug-ins are intended to provide different sorts of image processing filters. They take an input image as a parameter, perform some processing of its pixels and return new image a result. Depending on what the image processing routine does, the new image can be of different format (color to grayscale conversion, for example) and/or size (due to resize, rotation or other transformations, for example). Some image processing filter plug-ins can also do "in-place" image processing, which is changing the input image itself without creating new image as a result. This can be very useful when trying to save memory and increase performance.

Unlike video processing plug-ins, which are described further, image processing filter plug-ins are not supposed to have any internal run time state, which could affect what the plug-in does. The image processing function performed by these plug-ins is determined purely based on plug-ins properties. Applying these plug-ins twice to the same input image shall always produce same result (unless the plug-in uses some random number generator to add some randomness to the result image - like noise generation, some imaging effects, etc.).

Below are the methods exposed by image processing filter plug-ins:

  • boolean pluginObject:CanProcessInPlace( ) - Returns true if the plug-in can do in-place image processing or false otherwise.
  • boolean pluginObject:IsPixelFormatSupported( inputFormatName:string ) - Returns true if the plug-in accepts images of the specified pixel format or false otherwise. Note: for the list of possible pixel format names see imageObject:PixelFormat( ).
  • string pluginObject:GetOutputPixelFormat( inputFormatName:string ) - Returns pixel format name of an output image when plug-in is given an input image of the specified format. Most image processing plug-ins don't change pixel format of images. However there are plug-ins like Grayscale conversion for example, which do.
  • imageObject pluginObject:ProcessImage( inputImage:imageObject ) - Performs processing of the input image and returns new image as a result. The input image is left unchanged.
  • pluginObject:ProcessImageInPlace( inputImage:imageObject ) - Performs processing of the specified image if the plug-in supports in-place image processing. The method fails if the plug-in does not support it.

Note: If script's Main() function does only in-place image processing, then it is not required to set processed image back to host, since all processing was done on the image itself. However if new image was created during some image processing and it is required to be pushed back to host for further processing or display, then Host.SetImage() API must be used. If unsure, use the API always - it will not do harm even if setting same image back to host.

-- Create instances of two plug-ins
sepiaPlugin     = Host.CreatePluginInstance( 'Sepia' )
grayscalePlugin = Host.CreatePluginInstance( 'Grayscale' )

function Main( )
    -- Get image from the host application
    image = Host.GetImage( )

    -- Do in-place image processing
    sepiaPlugin:ProcessImageInPlace( image )
    -- Process an image getting new as a result
    newImage = grayscalePlugin:ProcessImage( image )

    -- Since we have new image, it should be set back to the host
    image = Host.SetImage( newImage )
end

Image processing 2 source filter plug-in interface

Supported starting from scripting API 3.

This plug-in's type is very similar to the above described image processing filter plug-in, but requires a second image in order to perform processing of the specified source image. The second image may be a mask, for example, if a plug-in intends to perform masking of some areas. It may be another image to add to or subtract from the source image. Or an image to merge, intersect, find difference with. It all depends on what the plug-in does. Note: the second image is never changed by the plug-in.

Below are the methods exposed by image processing 2 source filter plug-ins:

  • boolean pluginObject:CanProcessInPlace( ) - Returns true if the plug-in can do in-place image processing or false otherwise.
  • boolean pluginObject:IsPixelFormatSupported( inputFormatName:string ) - Returns true if the plug-in accepts images of the specified pixel format or false otherwise. Note: for the list of possible pixel format names see imageObject:PixelFormat( ).
  • string pluginObject:GetOutputPixelFormat( inputFormatName:string ) - Returns pixel format name of an output image when plug-in is given an input image of the specified format.
  • string pluginObject:GetSecondImageSupportedSize( ) - Returns supported size of a second image. Possible values are: "Any", "Equal", "Equal or Bigger" or "Equal or Smaller".
  • string pluginObject:GetSecondImageSupportedFormat( inputFormatName:string ) - Returns supported pixel format name of the second image for the given format name of the source image.
  • imageObject pluginObject:ProcessImage( inputImage:imageObject, inputImage2:imageObject ) - Performs processing of the input image and returns new image as a result. The input image is left unchanged.
  • pluginObject:ProcessImageInPlace( inputImage:imageObject, inputImage2:imageObject ) - Performs processing of the specified input image if the plug-in supports in-place image processing. The method fails if the plug-in does not support it.

Image processing plug-in interface

Supported starting from scripting API 3.

Although the type of these plug-ins sound very similar to the above described image processing filter plug-ins, these plug-ins are a bit different. Unlike filters, the image processing plug-ins never change provided image at all. All they do is perform some sort of image analysis/processing and calculate some value, which later can be retrieved in scripting, for example, as different read-only properties. As an example of this plug-ins' type can be mentioned Image Statistics plug-in, which calculates different statistical values, like RGB minimum, maximum, mean and standard deviation values, histograms, etc.

Below are the methods exposed by image processing plug-ins:

  • boolean pluginObject:IsPixelFormatSupported( inputFormatName:string ) - Returns true if the plug-in accepts images of the specified pixel format or false otherwise. Note: for the list of possible pixel format names see imageObject:PixelFormat().
  • pluginObject:ProcessImage( inputImage:imageObject ) - Performs processing of the input image and updates its read-only properties with new values based on the performed processing.

Video processing plug-in interface

Video processing plug-ins are very similar to image processing filter plug-ins, but have few differences however. Video processing plug-ins may or may not change input image. For example, plug-ins which save images into a video file don't do any changes to input images. Also video processing plug-ins are not intended to change images' size and pixel format, so these plug-ins always do in-place processing. The main difference however is that video processing plug-ins may have internal run time state, which determines what the plug-in does to the next image to be processed. For example, if a plug-in does video saving, then the internal state may contain open file/codec handles and other information for tracking video saving progress. Or, if a plug-in does some video processing aimed to detect/recognize/track something, then the internal state can be used for any variables needed to perform the task.

Below are the methods exposed by video processing plug-ins:

  • boolean pluginObject:IsReadOnlyMode( ) - Returns true if the plug-in does not do any changes to images while processing them or false otherwise.
  • boolean pluginObject:IsPixelFormatSupported( inputFormatName:string ) - Returns true if the plug-in accepts images of the specified pixel format or false otherwise. Note: for the list of possible pixel format names see imageObject:PixelFormat().
  • pluginObject:ProcessImage( inputImage:imageObject ) - Performs processing of the input image. The input image may or may not change depending on what the plug-in does and its current mode.
  • pluginObject:Reset( ) - Resets internal run time state of the plug-in bringing it back to the state as if it was just instantiated.

Image importing plug-in interface

Supported starting from scripting API 2.

Image importing plug-ins allow loading images from different file formats. Below is the interface provided by these plug-ins:

  • array pluginObject:SupportedExtensions( ) - Returns an array of supported files' extensions.
  • imageObject pluginObject:ImportImage( fileName:string ) - Loads image from the specified file and returns it as an image object.

Image exporting plug-in interface

Supported starting from scripting API 2.

Image exporting plug-ins allow saving images into different file formats. Below is the interface provided by these plug-ins:

  • array pluginObject:SupportedExtensions( ) - Returns an array of supported files' extensions.
  • array pluginObject:SupportedPixelFormats( ) - Returns an array of pixel formats supported by the image encoder.
  • pluginObject:ExportImage( fileName:string, imageToSave:imageObject ) - Saves image into the specified file.

Scripting API plug-in interface

Supported starting from scripting API 3.

The scripting API plug-ins don't provide any extra methods on top of the base plug-ins' interface. This type of plug-ins was created only to highlight the primary intention of the plug-ins, i.e. to provide APIs to be used from scripting. The main and the only purpose of these plug-ins is to expose some functions to be called from scripting by using plugin:CallFunction() method.

-- Sample of calling plug-in functions

-- Create an instances of image drawing plug-in
drawingPlugin = Host.CreatePluginInstance( 'ImageDrawing' )
-- Create an image to draw on
image = Image.Create( 640, 480, 'RGB24' )

-- Call different functions exposed by the plug-in
drawingPlugin:CallFunction( 'FillRectangle', image, { 20, 20 }, { 300, 250 }, '0000FF' )
drawingPlugin:CallFunction( 'DrawRectangle', image, { 20, 20 }, { 300, 250 }, 'FF0000' )

drawingPlugin:CallFunction( 'DrawText', image, 'Image Drawing',
                            { 50, 120 }, 'FFFFFF', '00000000' )

drawingPlugin:CallFunction( 'FillCircle', image, { 450, 300 }, 140, 'FFFFA0' )
drawingPlugin:CallFunction( 'DrawCircle', image, { 450, 300 }, 140, '80FF80' )

drawingPlugin:CallFunction( 'DrawLine', imageX, { 310, 300 }, { 590, 300 }, '80FF80' )
drawingPlugin:CallFunction( 'DrawLine', imageX, { 450, 160 }, { 450, 440 }, '80FF80' )

drawingPlugin:CallFunction( 'FillEllipse', image, { 150, 350 }, 100, 60, 'A0FFFF' )
drawingPlugin:CallFunction( 'DrawEllipse', image, { 150, 350 }, 100, 60, 'FF8080' )

Checking scripting API revision

To resolve which APIs are available, scripts may check "SCRIPTING_API_REVISION" global variable. Note: the variable itself was introduced since API revision 2.

-- Checking we have required API revision
if SCRIPTING_API_REVISION < 3 then
    error( "The script requires minimum API revision 3" )
end

-- If we want to avoid an error being generated for old scripting interface, where the
-- variable is not defined at all, we can also check it for "nil"
if ( SCRIPTING_API_REVISION == nil ) or ( SCRIPTING_API_REVISION < 3 ) then
    error( "The script requires minimum API revision 3" )
end

 

See also: