Switching effects at run time
Previous tutorial: Introduction to Scripting
The previous tutorial discussed support of scripting introduced in the 1.2.0 version of the Computer Vision Sandbox and how it can be used to make video processing graph more dynamic by changing different plug-ins' properties at runtime. With the plug-ins' set available so far scripting allows to create variety of different video processing effects.
This time we'll make a step further and let user to control which effect to run at the particular moment. To achieve this we may need a way to interact with environment and respond to changes happening in it. How can we do it from a script? What do we need for this? One possible way to do this is to use new plug-in type - device plug-in. And the plug-in we are going to use for this tutorial is called Led Keys, which is a simple device plug-in allowing to get state of system's LED keys - CapsLock, NumLock and ScrollLock. By checking current state of the LED keys, a script may decide which effect to run. And then it is up to user to run the script and play with the keys for controlling current effect.
Note: for the time being device plug-ins are not officially supported, nor documented. This is a subject for the next major release. The Led Keys is just a very simple plug-in, sort of a proof of concept. However it can already be used now to get even more fun with the scripting support.
So lets get to work. We'll write a Lua script containing 3 different video processing effects and we'll choose the one to run depending on which LED key is pressed. We'll not check for possible key combinations at this point leaving it as an exercise for those willing to have more than 3 effects. Remember that Lua scripts executed by the Lua scripting plug-in available with Computer Vision Sandbox contain two main parts: global space to perform any required initialization and a Main function to do the main bit of the script (video processing in the case script is running as part of video processing graph). So lets start with the initialization part and create instances of different image processing plug-ins we'll use for our effects (if something gets unclear, refer to the Lua Scripting API page). Note: we are allowed to define any extra functions to help us implementing the Main one.
-- Require Lua's math module local math = require "math" -- Create instances of plug-ins to use pixellatePlugin = Host.CreatePluginInstance( 'Pixellate' ) brightnessPlugin = Host.CreatePluginInstance( 'BrightnessCorrection' ) contrastPlugin = Host.CreatePluginInstance( 'ContrastCorrection' ) saturatePlugin = Host.CreatePluginInstance( 'Saturate' ) noisePlugin = Host.CreatePluginInstance( 'UniformAdditiveNoise' ) rotatePlugin = Host.CreatePluginInstance( 'RotateImage' ) borderPlugin = Host.CreatePluginInstance( 'RoundedBorder' ) sepiaPlugin = Host.CreatePluginInstance( 'Sepia' ) vignettingPlugin = Host.CreatePluginInstance( 'Vignetting' ) grainPlugin = Host.CreatePluginInstance( 'Grain' ) fuzzyBorderPlugin = Host.CreatePluginInstance( 'FuzzyBorder' ) -- Start values of some plug-ins' properties which will vary while effect is running pixelWidth = 1 pixelHeight = 1 brightnessFactor = 0 contrastFactor = 0 saturateBy = 0 noiseAmplitude = 20 rotateAngle = 0 borderWidth = 32 vignettingStartFactor = 80 vignettingEndFactor = 140 grainSpacing = 40 noiseAmplitude = 10 -- Set some other properties to predefined default value borderPlugin:SetProperty( 'borderColor', '000000' ) vignettingPlugin:SetProperty( 'decreaseSaturation', false ) vignettingPlugin:SetProperty( 'startFactor', vignettingStartFactor ) vignettingPlugin:SetProperty( 'endFactor', vignettingEndFactor ) grainPlugin:SetProperty( 'staticSeed', true ) grainPlugin:SetProperty( 'density', 0.5 ) fuzzyBorderPlugin:SetProperty( 'borderColor', '000000' ) fuzzyBorderPlugin:SetProperty( 'borderWidth', 32 ) fuzzyBorderPlugin:SetProperty( 'waviness', 8 ) fuzzyBorderPlugin:SetProperty( 'gradientWidth', 16 ) -- Other variables counter = 0 increasing = true maxPixelSize = 16 seed = 0 -- Make sure the specified value is in the specified range function CheckRange( value, min, max ) if value < min then value = min end if value > max then value = max end return value end
For the fist effect we'll do pixellation. But to make it more fun and dynamic, we'll keep changing width/height of pixels - first they grow, then they shrink, an repeat.
-- Perform 1st image processing effect function Effect1( image ) pixellatePlugin:ProcessImageInPlace( image ) -- Update properties if counter == 0 then if increasing then -- increase pixel width first, then height if pixelWidth == maxPixelSize then pixelHeight = pixelHeight + 1 if pixelHeight == maxPixelSize then increasing = false end else pixelWidth = pixelWidth + 1 end else -- decrease pixel width first, then height if pixelWidth == 1 then pixelHeight = pixelHeight - 1 if pixelHeight == 1 then increasing = true end else pixelWidth = pixelWidth - 1 end end pixellatePlugin:SetProperty( 'pixelWidth', pixelWidth ) pixellatePlugin:SetProperty( 'pixelHeight', pixelHeight ) end counter = ( counter + 1 ) % 5 return image end
The second effect looks more interesting though. It uses 6 different plug-ins and introduces randomness to all of them. As a result we get an effect of sort shaky unstable environment, where brightness, contrast and saturation always keep changing, amount of noise is also changing, the camera never looks stable and keeps shaking. Even the border can not calm down and get a fixed width.
-- Perform 2nd image processing effect function Effect2( image ) -- Randomize some properties of the plug-ins in use RandomizeEffect2( ) -- Apply image processing routines brightnessPlugin:ProcessImageInPlace( image ) contrastPlugin:ProcessImageInPlace( image ) saturatePlugin:ProcessImageInPlace( image ) noisePlugin:ProcessImageInPlace( image ) newImage = rotatePlugin:ProcessImage( image ) borderPlugin:ProcessImageInPlace( newImage ) return newImage end -- Modify plug-ins' properties randomly for the 2nd effect function RandomizeEffect2( ) -- change brightness factor brightnessFactor = CheckRange( brightnessFactor + math.random( ) / 5 - 0.1, -0.5, 0.5 ) brightnessPlugin:SetProperty( 'factor', brightnessFactor ) -- change contrast factor contrastFactor = CheckRange( contrastFactor + math.random( ) / 5 - 0.1, -0.5, 0.5 ) contrastPlugin:SetProperty( 'factor', contrastFactor ) -- change saturation saturateBy = CheckRange( saturateBy + math.random( ) * 4 - 2, -20, 20 ) saturatePlugin:SetProperty( 'saturateBy', saturateBy ) -- change noise level noiseAmplitude = CheckRange( noiseAmplitude + math.random( ) * 4 - 2, 10, 60 ) noisePlugin:SetProperty( 'amplitude', noiseAmplitude ) -- change rotation angle rotateAngle = CheckRange( rotateAngle + math.random( ) - 0.5, -5, 5 ) rotatePlugin:SetProperty( 'angle', rotateAngle ) -- change border width borderWidth = CheckRange( borderWidth + math.random( 3 ) - 2, 30, 35 ) borderPlugin:SetProperty( 'borderWidth', borderWidth ) end
And for the final effect we'll go back in time by making sort of an old style movie effect on a scratched film. For this will turn video into sepia colors, add vignetting so image gets darker further away from its center, then vertical grain for scratches, a bit of noise and finally decorate it with a fuzzy border.
-- Perform 3rd image processing effect function Effect3( image ) -- Randomize some properties of the plug-ins in use RandomizeEffect3( ) -- Apply image processing routines sepiaPlugin:ProcessImageInPlace( image ) vignettingPlugin:ProcessImageInPlace( image ) grainPlugin:ProcessImageInPlace( image ) noisePlugin:ProcessImageInPlace( image ) fuzzyBorderPlugin:ProcessImageInPlace( image ) return image end -- Modify plug-ins' properties randomly for the 3rd effect function RandomizeEffect3( ) -- change vignetting start/end factors vignettingStartFactor = CheckRange( vignettingStartFactor + math.random( 3 ) - 2, 70, 100 ) vignettingPlugin:SetProperty( 'startFactor', vignettingStartFactor ) vignettingEndFactor = CheckRange( vignettingEndFactor + math.random( 3 ) - 2, 120, 150 ) vignettingPlugin:SetProperty( 'endFactor', vignettingEndFactor ) -- change noise level noiseAmplitude = CheckRange( noiseAmplitude + math.random( ) * 4 - 2, 10, 30 ) noisePlugin:SetProperty( 'amplitude', noiseAmplitude ) -- change grain and every 5th frame counter = ( counter + 1 ) % 5 if counter == 0 then -- grain's seed value seed = seed + 1 grainPlugin:SetProperty( 'seedValue', seed ) -- grain's spacing grainSpacing = CheckRange( grainSpacing + math.random( 5 ) - 3, 30, 50 ) end end
Now it is time to test it all. Just to make sure it all works as expected, we can use a Main function like the below. It does not allow switching anything with LED keys yet, but it is better to test what we have so far before adding extra complexity.
-- Main function to be executed for every frame function Main( ) -- Get image to process image = Host.GetImage( ) -- Decide which effect to use newImage = Effect1( image ) --newImage = Effect2( image ) --newImage = Effect3( image ) -- Set the new image back to host Host.SetImage( newImage ) end
And now is the final bit we wanted to get to from the very beginning. We have all the 3 effects working and tested. Those are based on a number of different image processing plug-ins. So lets create an instance of the final plug-in we need, the Led Keys device plug-in, and use its properties to discover current state of the system's LED keys. Note: device plug-ins must be connected to devices they manage, so don't miss the Connect() API call.
ledKeysPlugin = Host.CreatePluginInstance( 'LedKeys' ) -- Connect LedKeys plug-in, so we can query status of CapsLock, NumLock and ScrolLock ledKeysPlugin:Connect( ) -- Main function to be executed for every frame function Main( ) -- Get image to process image = Host.GetImage( ) -- Decide which effect to use if ledKeysPlugin:GetProperty( 'capsLock' ) then newImage = Effect1( image ) elseif ledKeysPlugin:GetProperty( 'numLock' ) then newImage = Effect2( image ) elseif ledKeysPlugin:GetProperty( 'scrollLock' ) then newImage = Effect3( image ) else newImage = image end -- Set the new image back to host Host.SetImage( newImage ) end
Looks good, isn't it? And if add the Virtual Camera Push plug-in into the video processing graph right after the Lua Scripting plug-in so we can push video into the CVSandbox Virtual Camera, then we can make our Skype calls even more fun and entertaining.
Well, time to wrap it up. As we can see there is much more to scripting then just putting image processing plug-ins together to get some effects. Using device plug-ins scripts may interact with environment and respond to its changes. This demo shows just how to make use of system's LED keys. But another device could be an Arduino board, for example, and so script could make use of any other sensors connected to it - push buttons, potentiometers, light sensors, temperature sensors, etc. But, we'll get back to it when support of device plug-ins is officially released.
Note: for a quick start here is a link to the complete script described in this tutorial.
Next tutorial: Processing sub images