Image slide show combined with video

Previous tutorial: Processing sub images

 

This tutorial will discuss how to put image files from a specified folder on top of an incoming video (or the other way around depending on what should become the main scene) and iterate through them when user presses a button. To put images on top of a video we'll use previously mentioned PutImage() API. For user interaction we'll utilize LedKeys plug-in described in the Switching effects tutorial. And for loading images we'll need something new - image importing plug-ins. Note: the Computer Vision Sandbox package version 1.2.1 is distributed with two image importing plug-ins: JpegImporter and PngImporter, which are aimed for loading JPEG and PNG images correspondingly. We'll use the JpegImporter for now.

The first part of the script has almost nothing to do with Computer Vision Sandbox, its plug-ins and APIs exposed by Lua scripting plug-in (apart from checking API revision). The Lua code below just collects a list of JPEG files from the specified folder and puts them into array for later use. 

-- Make sure we have required APIs exposed
if SCRIPTING_API_REVISION == nil then
    error( 'The script requires minimum API revision 2' )
end

local io = require "io"

-- Folder to look for *.jpg files in
folder  = [[C:\Users\Andrew\Pictures\]]

-- Get the list of files to browse (works on Window only)
prog    = 'dir "' .. folder .. '*.jpg" /b /a-d'
filesIt = io.popen( prog ):lines( )

-- Build array out of iterator function
files = { }
for file in filesIt do
    files[#files + 1] = file
end
filesCount = #files

-- Check number of files
if filesCount == 0 then
    error( "Did not find any JPEG files (*.jpg) in the specified folder" )
end

The next part of the code is simple enough as well - just creating plug-ins to use and initializing some variables. As it was mentioned before, the JpegImporter plug-in will be used to load JPEG images from the specified folder, LedKeys to interact with user, ResizeImage and GrayscaleToRgb to manipulate images (resize them to required size and convert grayscale JPEG's to RGB in case we get such). 

-- Create instances of plug-ins to use
jpegImporterPlugin = Host.CreatePluginInstance( 'JpegImporter' )
grayToRgbPlugin    = Host.CreatePluginInstance( 'GrayscaleToRgb' )
resizeImagePlugin  = Host.CreatePluginInstance( 'ResizeImage' )
ledKeysPlugin      = Host.CreatePluginInstance( 'LedKeys' )

-- Set bilinear interpolation (1). Nearest neighbour (0) can be also used to make it faster,
-- but reduces quality
resizeImagePlugin:SetProperty( 'interpolation', 1 )

lastWidth   = 0
lastHeight  = 0
currentFile = 1

-- Loaded image and its resized version
loadedImage        = nil
resizedLoadedImage = nil

-- Connect the LedKeys device plug-in and get NumLock status
ledKeysPlugin:Connect( )
prevNumLockState = ledKeysPlugin:GetProperty( 'numLock' )

Now lets get to implementation of the Main function. The first thing to point is that our pictures' slide show is protected by Caps Lock key - if it is pressed, the slide show is on; if it is not pressed - then normal video is displayed as it comes from a video source. The second point to note is that Num Lock key is used to switch images - when user presses Num Lock (does not mater if it goes On or Off, all we need is a change in its state) a new image is loaded. 

-- Main function to be executed for every frame
function Main( )
    -- Get image to process
    image = Host.GetImage( )

    width  = image:Width( )
    height = image:Height( )

    smallWidth  = width  / 4;
    smallHeight = height / 4;

    if ledKeysPlugin:GetProperty( 'capsLock' ) then

        -- Check if we need to switch to another image
        if prevNumLockState ~= ledKeysPlugin:GetProperty( 'numLock' ) then
            currentFile = currentFile + 1
            if currentFile > filesCount then
                currentFile = 1
            end

            if loadedImage ~= nil then
                loadedImage:Release( )
                loadedImage = nil
            end

            if resizedLoadedImage ~= nil then
                resizedLoadedImage:Release( )
                resizedLoadedImage = nil
            end
        end

        prevNumLockState = ledKeysPlugin:GetProperty( 'numLock' )

Now a bit of image manipulation: 1) a new image is loaded if required; 2) grayscale images get converted to color, so we can put them on top of incoming video (we rely on the fact video frames come as 24 bpp color images); 3) then loaded image is resized to the required size. This all is accompanied by some extra code taking care of releasing images, etc. 

-- Note: ignore indention here - it is all under "if capsLock ON" ...

-- Load image if required
if loadedImage == nil then
    loadedImage = jpegImporterPlugin:ImportImage( folder .. files[currentFile] )
    
    -- If we got grayscale JPEG, convert it to RGB
    if loadedImage:PixelFormat( ) == 'Gray8' then
        tempImage = grayToRgbPlugin:ProcessImage( loadedImage )
        loadedImage:Release( )
        loadedImage = tempImage
    end
end

-- Make sure video size did not change
if ( lastWidth ~= width ) or ( lastHeight ~= height ) then
    if resizedLoadedImage ~= nil then
        resizedLoadedImage:Release( )
        resizedLoadedImage = nil
    end
end

-- Determine the size to resize loaded image to
resizeWidth  = smallWidth
resizeHeight = smallHeight

if ledKeysPlugin:GetProperty( 'scrollLock' ) then
    resizeWidth  = width
    resizeHeight = height
end

-- Release any image of the wrong size
if ( resizedLoadedImage ~= nil ) and
   ( ( resizeWidth  ~= resizedLoadedImage:Width( ) ) or
     ( resizeHeight ~= resizedLoadedImage:Height( ) ) ) then
    
    resizedLoadedImage:Release( )
    resizedLoadedImage = nil
end

-- Create new resized image
if resizedLoadedImage == nil then
    resizeImagePlugin:SetProperty( 'width',  resizeWidth )
    resizeImagePlugin:SetProperty( 'height', resizeHeight )
    resizedLoadedImage = resizeImagePlugin:ProcessImage( loadedImage )
end

Now we are ready to complete it all and put the loaded image on top of the incoming video. However, before doing that, lets get a bit more interactivity. We'll makes use of yet another Led Key. If Scroll Lock is Off, we'll do it the way we planned - we'll put loaded image on top of the video in the top right corner. Below is the code for it and the way it looks (the toy fish cames from video, while the bird cames from a picture made on some holiday). 

if ledKeysPlugin:GetProperty( 'scrollLock' ) == false then
    -- put resized loaded image in the top-right corner
    image:PutImage( resizedLoadedImage, width - smallWidth, 0 )
else
    ...
end

And now lets handle the case when Scroll Lock is On. In this case we'll do everything the other way around - we'll put video in the top-right corner of the loaded image, which now occupies the entire scene. So by changing state of the Scroll Lock key, we can switch between picture "preview mode" and "slide show mode". 

if ledKeysPlugin:GetProperty( 'scrollLock' ) == false then
    ...
else
    -- the loaded image takes the entire frame, but video goes to the top-right corner

    -- resize video frame
    resizeImagePlugin:SetProperty( 'width',  smallWidth )
    resizeImagePlugin:SetProperty( 'height', smallHeight )
    resizedVideo = resizeImagePlugin:ProcessImage( image )

    resizedLoadedImage:PutImage( resizedVideo, width - smallWidth, 0 )

    -- Put new image back to host
    Host.SetImage( resizedLoadedImage )
    
    -- Release the resized video frame
    resizedVideo:Release( )
end

lastWidth  = width
lastHeight = height

All the described above can be change to avoid using any "Led Keys" at all. Instead the script can be changed to make a fully automated slide show, when pictures change after X number of seconds/frames. Where to use it all? Well, there are number of applications when it might be required to combine video with a static picture for decoration, displaying logo, something else. Combining it with the Virtual video camera described previously, may bring more fun to your Skype conversations, allowing you to show remote party some of your pictures and at the same time staying visible as well.

Note: for a quick start here is a link to the complete script described in this tutorial.

 

Next tutorial: Releasing resources and garbage collection