Sandbox scripting threads

Previous tutorial: Device plug-ins


The previous article described new types of plug-ins aimed for communication with different devices and how this can be done by using scripting engine. Interaction with different devices can be done from the same scripts as those used to perform some video processing. For example, a video processing routine may change depending on values of some sensors. Or device's actuators can be controlled based on the results of performed image processing. Mixing video processing and communication with devices opens possibility for a great range of different applications.

In many cases, however, it is preferred to put communication with devices into separate scripts instead of doing it from video processing scripts. There are number of reasons for that. First, there may not be relation to performed video processing at all. For example, a pan/tilt device may move camera at certain time intervals, which don't depend on results of video processing algorithms. Or, robot's movement can be controlled based on inputs from another device. Second, very often it is preferred to complete video processing as soon as possible, so that video source does not get blocked. Communication with some devices, however, may involve certain delays caused by connection speed, protocols in use, etc. Another reason could be requirement to interact with certain devices at time intervals, which are not based on video source's frame rate, i.e. have more frequent interactions with some devices and less frequent with others.

To address the need of running some scripts independent from video processing, the 2.0.0 version of Computer Vision Sandbox introduced the concept of sandbox scripting threads. Sandbox wizard now allows not only configuring which video processing steps to run for each camera within a sandbox, but also create additional threads, which run specified scripts at set time intervals. For example, the screenshot below demonstrates a possible set-up for controlling the PiRex robot. The first thread runs a script for controlling robot's motors based of game pad's input. To make the robot responsive enough, the thread runs control script at 10 milliseconds intervals. The second thread runs a different script, which queries distance measurements provided by robot's ultrasonic sensor. We chose it to run 10 times a second, i.e. at 100 milliseconds intervals.

The scripts running within sandbox threads have very similar structure to those used to perform video processing on camera's images. They have a global section and a Main function. The global section is executed once, when sandbox gets started (before starting any video sources). And the Main function is executed again and again at configured time intervals.

Let's have a look at potential implementation of the scripts used for the above shown sandbox threads. The first script does robot's control - changes motors' power based on game pad's input. It has nothing to do with the video coming from robot's camera and we want to run it at a higher rate than camera's FPS. Looks like a perfect candidate to run on its own in a sandbox thread. All it does is reading values of game pad's axes (using two different axes for robot's left/right motors), converting those into motors' power values and sending them to the robot so it performs desired movement.

local math = require 'math'

gamepadPlugin = Host.CreatePluginInstance( 'Gamepad' )
pirexPlugin   = Host.CreatePluginInstance( 'PiRexBot' )

prevLeftPower  = 1000
prevRightPower = 1000

-- Configure gamepad and connect to it
gamepadPlugin:SetProperty( 'deviceId', 0 )
gamepadPlugin:Connect( )

-- Configure PiRex Bot and connect to it
pirexPlugin:SetProperty( 'address', '' )
pirexPlugin:Connect( )

function Main( )
    axesValues = gamepadPlugin:GetProperty( 'axesValues' )

    -- Pulling gamepad's axis up result in -100, down: 100
    -- So need to invert it here to get something making sense
    leftPower  = 0 - math.floor( axesValues[2] * 100 )
    rightPower = 0 - math.floor( axesValues[3] * 100 )

    -- Set motors' power
    if ( math.abs( prevLeftPower - leftPower ) ) then
        pirexPlugin:SetProperty( 'leftMotor', leftPower )

    if ( math.abs( prevRightPower - rightPower ) ) then
        pirexPlugin:SetProperty( 'rightMotor', rightPower )

    -- Remember motors' power
    prevLeftPower  = leftPower
    prevRightPower = rightPower

The second script we have runs at 100 milliseconds intervals and is used to read distance measurements provided by robot's ultrasonic sensor. There is not much we'll do about it, but just display it to user directly on the video coming from robot's camera. This requires some image processing (drawing) for displaying the distance to obstacles, which means we could put the code for reading the sensor into the script doing camera's video processing. However, as mentioned before, sensor reading may cause certain delays and we don't really want to introduce those into video processing. So, we'll separate sensor reading and measurement displaying into two scripts, which communicate by using host variables.

local string = require 'string'

pirexPlugin = Host.CreatePluginInstance( 'PiRexBot' )

-- Configure PiRex Bot and connect to it
pirexPlugin:SetProperty( 'address', '' )
pirexPlugin:Connect( )

function Main( )
    -- Get distance to obstacles in front of the robot
    distance = pirexPlugin:GetProperty( 'obstacleDistance' )

    Host.SetVariable( 'obstacleDistance', string.format( '%.2f', distance ) )

As we can see from above, the script only reads distance measurements and puts those into a host variable - nothing more. Obviously, this will not display anything to user, but this is where video processing script comes into play. Among other things we may want to do with images coming from robot's camera, we can also output the distance measurement, which can be retrieved from the host variable.

drawing = Host.CreatePluginInstance( 'ImageDrawing' )

function Main( )
    image    = Host.GetImage( )
    distance = Host.GetVariable( 'obstacleDistance' )
    -- ... Perform any image processing we wish to ...
    -- Distance to obstacles in front of robot
    drawing:CallFunction( 'DrawText', image, 'Distance : ' .. distance,
                          { 10, 10 }, '00FF00', '00000000' )

The above use case demonstrates usage of sandbox threads and how those can be used to perform certain actions at configured time intervals. All scripts (threading or video processing) running within a sandbox can communicate by setting/reading host variables. This may allow different scenarios. A video processing routine can be driven by reading some sensors. However, an opposite can be done as well, i.e. a video processing script may set some variables based on the results of applied algorithms and then a threading script can read those variables and drive some device's actuators. This all allows building more interesting and interactive applications, where video processing can be affected by real world events and the other way around.


Next tutorial: Sandbox variables monitor