Switching effects at run time

Previous tutorial: Introduction to Scripting

 

The previous tutorial discussed support of scripting introduced in the 1.2.0 version of the Computer Vision Sandbox and how it can be used to make video processing graph more dynamic by changing different plug-ins' properties at runtime. With the plug-ins' set available so far scripting allows to create variety of different video processing effects.

This time we'll make a step further and let user to control which effect to run at the particular moment. To achieve this we may need a way to interact with environment and respond to changes happening in it. How can we do it from a script? What do we need for this? One possible way to do this is to use new plug-in type - device plug-in. And the plug-in we are going to use for this tutorial is called Led Keys, which is a simple device plug-in allowing to get state of system's LED keys - CapsLock, NumLock and ScrollLock. By checking current state of the LED keys, a script may decide which effect to run. And then it is up to user to run the script and play with the keys for controlling current effect.

Note: for the time being device plug-ins are not officially supported, nor documented. This is a subject for the next major release. The Led Keys is just a very simple plug-in, sort of a proof of concept. However it can already be used now to get even more fun with the scripting support.

So lets get to work. We'll write a Lua script containing 3 different video processing effects and we'll choose the one to run depending on which LED key is pressed. We'll not check for possible key combinations at this point leaving it as an exercise for those willing to have more than 3 effects. Remember that Lua scripts executed by the Lua scripting plug-in available with Computer Vision Sandbox contain two main parts: global space to perform any required initialization and a Main function to do the main bit of the script (video processing in the case script is running as part of video processing graph). So lets start with the initialization part and create instances of different image processing plug-ins we'll use for our effects (if something gets unclear, refer to the Lua Scripting API page). Note: we are allowed to define any extra functions to help us implementing the Main one.

-- Require Lua's math module
local math = require "math"

-- Create instances of plug-ins to use
pixellatePlugin  = Host.CreatePluginInstance( 'Pixellate' )

brightnessPlugin  = Host.CreatePluginInstance( 'BrightnessCorrection' )
contrastPlugin    = Host.CreatePluginInstance( 'ContrastCorrection' )
saturatePlugin    = Host.CreatePluginInstance( 'Saturate' )
noisePlugin       = Host.CreatePluginInstance( 'UniformAdditiveNoise' )
rotatePlugin      = Host.CreatePluginInstance( 'RotateImage' )
borderPlugin      = Host.CreatePluginInstance( 'RoundedBorder' )

sepiaPlugin       = Host.CreatePluginInstance( 'Sepia' )
vignettingPlugin  = Host.CreatePluginInstance( 'Vignetting' )
grainPlugin       = Host.CreatePluginInstance( 'Grain' )
fuzzyBorderPlugin = Host.CreatePluginInstance( 'FuzzyBorder' )

-- Start values of some plug-ins' properties which will vary while effect is running
pixelWidth       = 1
pixelHeight      = 1

brightnessFactor = 0
contrastFactor   = 0
saturateBy       = 0
noiseAmplitude   = 20
rotateAngle      = 0
borderWidth      = 32

vignettingStartFactor = 80
vignettingEndFactor   = 140
grainSpacing          = 40
noiseAmplitude        = 10

-- Set some other properties to predefined default value
borderPlugin:SetProperty( 'borderColor', '000000' )
vignettingPlugin:SetProperty( 'decreaseSaturation', false )
vignettingPlugin:SetProperty( 'startFactor', vignettingStartFactor )
vignettingPlugin:SetProperty( 'endFactor', vignettingEndFactor )
grainPlugin:SetProperty( 'staticSeed', true )
grainPlugin:SetProperty( 'density', 0.5 )
fuzzyBorderPlugin:SetProperty( 'borderColor', '000000' )
fuzzyBorderPlugin:SetProperty( 'borderWidth', 32 )
fuzzyBorderPlugin:SetProperty( 'waviness', 8 )
fuzzyBorderPlugin:SetProperty( 'gradientWidth', 16 )

-- Other variables
counter      = 0
increasing   = true
maxPixelSize = 16
seed         = 0

-- Make sure the specified value is in the specified range
function CheckRange( value, min, max )
    if value < min then value = min end
    if value > max then value = max end
    return value
end

For the fist effect we'll do pixellation. But to make it more fun and dynamic, we'll keep changing width/height of pixels - first they grow, then they shrink, an repeat.

-- Perform 1st image processing effect
function Effect1( image )

    pixellatePlugin:ProcessImageInPlace( image )

    -- Update properties
    if counter == 0 then
        if increasing then
            -- increase pixel width first, then height
            if pixelWidth == maxPixelSize then
                pixelHeight = pixelHeight + 1
                if pixelHeight == maxPixelSize then
                    increasing = false
                end
            else
                pixelWidth = pixelWidth + 1
            end
        else
            -- decrease pixel width first, then height
            if pixelWidth == 1 then
                pixelHeight = pixelHeight - 1
                if pixelHeight == 1 then
                    increasing = true
                end
            else
                pixelWidth = pixelWidth - 1
            end
        end

        pixellatePlugin:SetProperty( 'pixelWidth', pixelWidth )
        pixellatePlugin:SetProperty( 'pixelHeight', pixelHeight )
    end

    counter = ( counter + 1 ) % 5

    return image
end

The second effect looks more interesting though. It uses 6 different plug-ins and introduces randomness to all of them. As a result we get an effect of sort shaky unstable environment, where brightness, contrast and saturation always keep changing, amount of noise is also changing, the camera never looks stable and keeps shaking. Even the border can not calm down and get a fixed width.

-- Perform 2nd image processing effect
function Effect2( image )
    -- Randomize some properties of the plug-ins in use
    RandomizeEffect2( )
    -- Apply image processing routines
    brightnessPlugin:ProcessImageInPlace( image )
    contrastPlugin:ProcessImageInPlace( image )
    saturatePlugin:ProcessImageInPlace( image )
    noisePlugin:ProcessImageInPlace( image )
    newImage = rotatePlugin:ProcessImage( image )
    borderPlugin:ProcessImageInPlace( newImage )

    return newImage
end

-- Modify plug-ins' properties randomly for the 2nd effect
function RandomizeEffect2( )
    -- change brightness factor
    brightnessFactor = CheckRange( brightnessFactor + math.random( ) / 5 - 0.1, -0.5, 0.5 )
    brightnessPlugin:SetProperty( 'factor', brightnessFactor )
    -- change contrast factor
    contrastFactor = CheckRange( contrastFactor + math.random( ) / 5 - 0.1, -0.5, 0.5 )
    contrastPlugin:SetProperty( 'factor', contrastFactor )
    -- change saturation
    saturateBy = CheckRange( saturateBy + math.random( ) * 4 - 2, -20, 20 )
    saturatePlugin:SetProperty( 'saturateBy', saturateBy )
    -- change noise level
    noiseAmplitude = CheckRange( noiseAmplitude + math.random( ) * 4 - 2, 10, 60 )
    noisePlugin:SetProperty( 'amplitude', noiseAmplitude )
    -- change rotation angle
    rotateAngle = CheckRange( rotateAngle + math.random( ) - 0.5, -5, 5 )
    rotatePlugin:SetProperty( 'angle', rotateAngle )
    -- change border width
    borderWidth = CheckRange( borderWidth + math.random( 3 ) - 2, 30, 35 )
    borderPlugin:SetProperty( 'borderWidth', borderWidth )
end

And for the final effect we'll go back in time by making sort of an old style movie effect on a scratched film. For this will turn video into sepia colors, add vignetting so image gets darker further away from its center, then vertical grain for scratches, a bit of noise and finally decorate it with a fuzzy border.

-- Perform 3rd image processing effect
function Effect3( image )
    -- Randomize some properties of the plug-ins in use
    RandomizeEffect3( )
    -- Apply image processing routines
    sepiaPlugin:ProcessImageInPlace( image )
    vignettingPlugin:ProcessImageInPlace( image )
    grainPlugin:ProcessImageInPlace( image )
    noisePlugin:ProcessImageInPlace( image )
    fuzzyBorderPlugin:ProcessImageInPlace( image )

    return image
end

-- Modify plug-ins' properties randomly for the 3rd effect
function RandomizeEffect3( )
    -- change vignetting start/end factors
    vignettingStartFactor = CheckRange( vignettingStartFactor + math.random( 3 ) - 2, 70, 100 )
    vignettingPlugin:SetProperty( 'startFactor', vignettingStartFactor )
    vignettingEndFactor = CheckRange( vignettingEndFactor + math.random( 3 ) - 2, 120, 150 )
    vignettingPlugin:SetProperty( 'endFactor', vignettingEndFactor )

    -- change noise level
    noiseAmplitude = CheckRange( noiseAmplitude + math.random( ) * 4 - 2, 10, 30 )
    noisePlugin:SetProperty( 'amplitude', noiseAmplitude )

    -- change grain and every 5th frame
    counter = ( counter + 1 ) % 5
    if counter == 0 then
        -- grain's seed value
        seed = seed + 1
        grainPlugin:SetProperty( 'seedValue', seed )
        -- grain's spacing
        grainSpacing = CheckRange( grainSpacing + math.random( 5 ) - 3, 30, 50 )
    end
end

Now it is time to test it all. Just to make sure it all works as expected, we can use a Main function like the below. It does not allow switching anything with LED keys yet, but it is better to test what we have so far before adding extra complexity.

-- Main function to be executed for every frame
function Main( )
    -- Get image to process
    image = Host.GetImage( )

    -- Decide which effect to use
    newImage = Effect1( image )
    --newImage = Effect2( image )
    --newImage = Effect3( image )

    -- Set the new image back to host
    Host.SetImage( newImage )
end

And now is the final bit we wanted to get to from the very beginning. We have all the 3 effects working and tested. Those are based on a number of different image processing plug-ins. So lets create an instance of the final plug-in we need, the Led Keys device plug-in, and use its properties to discover current state of the system's LED keys. Note: device plug-ins must be connected to devices they manage, so don't miss the Connect() API call.

ledKeysPlugin = Host.CreatePluginInstance( 'LedKeys' )

-- Connect LedKeys plug-in, so we can query status of CapsLock, NumLock and ScrolLock
ledKeysPlugin:Connect( )

-- Main function to be executed for every frame
function Main( )
    -- Get image to process
    image = Host.GetImage( )

    -- Decide which effect to use
    if ledKeysPlugin:GetProperty( 'capsLock' ) then
        newImage = Effect1( image )
    elseif ledKeysPlugin:GetProperty( 'numLock' ) then
        newImage = Effect2( image )
    elseif ledKeysPlugin:GetProperty( 'scrollLock' ) then
        newImage = Effect3( image )
    else
        newImage = image
    end

    -- Set the new image back to host
    Host.SetImage( newImage )
end

Looks good, isn't it? And if add the Virtual Camera Push plug-in into the video processing graph right after the Lua Scripting plug-in so we can push video into the CVSandbox Virtual Camera, then we can make our Skype calls even more fun and entertaining.

Well, time to wrap it up. As we can see there is much more to scripting then just putting image processing plug-ins together to get some effects. Using device plug-ins scripts may interact with environment and respond to its changes. This demo shows just how to make use of system's LED keys. But another device could be an Arduino board, for example, and so script could make use of any other sensors connected to it - push buttons, potentiometers, light sensors, temperature sensors, etc. But, we'll get back to it when support of device plug-ins is officially released.

Note: for a quick start here is a link to the complete script described in this tutorial.

 

Next tutorial: Processing sub images

Copyright © 2015-2019 Andrew Kirillov