Code for YUV video stream processing. This code extends code from the MJPEG tools and requires the mjpeg libraries. If you have difficulty compiling these contact me, mjpeg0 at silicontrip dot org.

You will need the mjpeg libraries to compile the yuv tools. information on the mjpeg libraries and download.

The Libav tools require the libavcodec library which is part of the ffmpeg tools

Contents

libav-bitrate
libav2yuv
libavmux
yuvaddetect
yuvadjust
yuvafps
yuvaifps
yuvCIFilter
yuvconvolve
yuvcrop
yuvdeinterlace
yuvdiag
yuvdiff
yuvfade
yuvGENERIC
yuvilace
yuvrfps
yuvwater

libav-bitrate

libav-bitrate.c

Bitrate graph producer

Produces an ASCII file suitable for plotting in gnuplot of the bitrate per frame. Also includes a rolling average algorithm that helps smooth out differences between I frame, P frame and B frames. Constant bitrate is not so constant. Requires about 50 frames rolling average window to get an idea of the average bitrate.

using a post processing tool such as octave to average the graph may be useful

RESULTS

VBR example

CBR example

libav2yuv

libav2yuv.c

Anything to YUV

Tested the EDL mode of this program. Discovered what appears to be a bug in the libav library in which frames are unexpectedly dropped. I have attempted to work around the bug, however it appears to be filetype specific. Currently m2v files work accurately. Use this with the updated yuvdiag tool to stamp your video for EDL file creation. Or use a frame accurate video player.

This is a very simple generic anything to yuv tool, using the libav library from ffmpeg. It supports any video file that libav supports (which includes WMV3s now!!!) it can be used to replace the -vo yuv4mpeg option of mplayer or the -f yuv4mpegpipe option of ffmpeg. The tool can also override yuv header values, in the case of incorrect aspect ratio or interlacing etc. And will perform simple chroma subsampling conversion, if the source file is not in the desired chroma subsampling mode.

This release now includes an EDL file parser, for joining or editing files. This is a first working version is not optimised and may contain bugs. I hope to have more testing over the next few months. See sample EDL file below.

Now uses sws_scaler for chroma resampling rather than the deprecated img_convert, So will compile with the newer version of FFMPEG. Also have provision for decode_audio3 and decode_video2 which has appeared in the latest ffmpeg snapshot.

I use this regularily for file conversions, so I hope the code is quite mature, with a thorough set of options and help. Any bugs should be picked up quickly.

As I do want this to be a one program fits all for yuv video conversion any bugs or issues you have with the software I would like to hear about so I can make this the best software.

EXAMPLE

libav2yuv strangefile.avi | y4m-yuvfilter | ffmpeg -f yuv4mpegpipe -i - -vcodec whatever wantedfile.avi

HISTORY

#     [num]  [duration]  [srcIn]  [srcOut]  [recIn]  [recOut]
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_01.mpg VA C 0:0:0:0 0:4:9:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_02.mpg VA C 0:0:0:0 0:3:39:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_03.mpg VA C 0:0:0:0 0:2:38:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_04.mpg VA C 0:0:0:0 0:3:9:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_05.mpg VA C 0:0:0:0 0:4:39:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_06.mpg VA C 0:0:0:0 0:4:9:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_07.mpg VA C 0:0:0:0 0:2:9:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_08.mpg VA C 0:0:0:0 0:3:9:0
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_09.mpg VA C 0:0:0:0 0:3:38:15
/Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_10.mpg VA C 0:0:0:0 0:5:46:0

The fields are:

libavmux

libavmux.c

Elementary stream to Container MUXER/DEMUXER

This is a very simple avi/mov/mpg demuxer and muxer.

The demuxer works quite well, however the muxing part still has bugs and has not correctly produced a working file. This is still a work in progress.

yuvaddetect

yuvaddetect.c

Advertisement detection

INCOMPLETE

Currently only produces a text file (which can be graphed by gnuplot) of the average lightness of the sourceframes. The idea is that there is a black fade before and after advertisements. To produce chapter markers for DVDs from TV.

RESULTS

From looking at this above graph it appears that this program is doing a frame difference, rather than averaging luma. Which is the same function as yuvdiff. Now we can quite easily see scene changes, shown as sharp spikes. At about 810 and 1500

We can also tell that the video has gone through a frame rate doubling by frame dupliation. Shown by the high frequency component.

I should move this image down to YUV diff

yuvadjust

yuvadjust.c

Adjust video

Allows changes such as brightness, contrast, saturation, hue shift and hue rotation. Supports negative values for inversion.

RESULTS


yuvafps

yuvafps.c

Averaging Frame rate Converter

A linear frame averaging, frame rate converter. Does a better job than the frame dropping/duplicating converters.

I did think that this no longer needed maintaining since yuvfps now has a weighted average conversion. However I find this programs conversion much nicer than yuvfps. This will work with converting very slow or very fast frame rates and averages all frames that overlap.

It supports interlaced and progressive streams, and can force a conversion from progressive to interlace (although I'm not sure if it properly handles progressive to interlaced chroma)

yuvaifps.c

Averaging Integer based Frame Rate Converter

I have re-written this to use integers rather than floats, appears to increase speed by about 3 times.

BUGS

It may overflow with long files (greater than about 80 minutes) and large prime number frame rate values. However these values are not very common. I've tried to detect this occurance, but it simply reports an error on stderr and continues with a corrupted stream.

yuvaifps

yuvaifps.c

see yuvafps

yuvCIFilter

yuvCIFilter.m

YUV Core Image Filter

This allows Core Image filters to operate on yuv streams.

Obviously this program only works under OSX.

Example

To choose a Core Image filter use the -f option: | yuvCIFilter -f CILineScreen |

Default filter settings are shown: 

{CILineScreen {
    inputAngle = 0;
    inputCenter = [150 150];
    inputImage = ;
    inputSharpness = 0.7;
    inputWidth = 6;
}}

Any of the filter settings can be changed using the -i option: yuvCIFilter -f CILineScreen -i inputAngle=45.0

Multiple settings are separated by a comma: yuvCIFilter -f CILineScreen -i inputAngle=45.0,inputWidth=12.0

Some settings require a space and will need to be quoted on the command line.

A list of filters is obtained by giving an invalid filtername to the -f option (such as help) 

CIAdditionCompositing
CIAffineClamp
CIAffineTile
CIAffineTransform
CIAreaAverage
CIAreaHistogram
CIAreaMaximum
CIAreaMaximumAlpha
CIAreaMinimum
CIAreaMinimumAlpha
CIBarsSwipeTransition
CIBlendWithMask
CIBloom
CIBoxBlur
CIBumpDistortion
CIBumpDistortionLinear
CICheckerboardGenerator
CICircleSplashDistortion
CICircularScreen
CICircularWrap
CICMYKHalftone
CIColorBlendMode
CIColorBurnBlendMode
CIColorControls
CIColorCube
CIColorDodgeBlendMode
CIColorInvert
CIColorMap
CIColorMatrix
CIColorMonochrome
CIColorPosterize
CIColumnAverage
CIComicEffect
CIConstantColorGenerator
CICopyMachineTransition
CICrop
CICrystallize
CIDarkenBlendMode
CIDifferenceBlendMode
CIDiscBlur
CIDisintegrateWithMaskTransition
CIDisplacementDistortion
CIDissolveTransition
CIDotScreen
CIEdges
CIEdgeWork
CIEightfoldReflectedTile
CIExclusionBlendMode
CIExposureAdjust
CIFalseColor
CIFlashTransition
CIFourfoldReflectedTile
CIFourfoldRotatedTile
CIFourfoldTranslatedTile
CIGammaAdjust
CIGaussianBlur
CIGaussianGradient
CIGlassDistortion
CIGlassLozenge
CIGlideReflectedTile
CIGloom
CIHardLightBlendMode
CIHatchedScreen
CIHeightFieldFromMask
CIHexagonalPixellate
CIHoleDistortion
CIHueAdjust
CIHueBlendMode
CIKaleidoscope
CILanczosScaleTransform
CILenticularHaloGenerator
CILightenBlendMode
CILinearGradient
CILineOverlay
CILineScreen
CILuminosityBlendMode
CIMaskToAlpha
CIMaximumComponent
CIMaximumCompositing
CIMedianFilter
CIMinimumComponent
CIMinimumCompositing
CIModTransition
CIMotionBlur
CIMultiplyBlendMode
CIMultiplyCompositing
CINoiseReduction
CIOpTile
CIOverlayBlendMode
CIPageCurlTransition
CIParallelogramTile
CIPerspectiveTile
CIPerspectiveTransform
CIPinchDistortion
CIPixellate
CIPointillize
CIRadialGradient
CIRandomGenerator
CIRippleTransition
CIRowAverage
CISaturationBlendMode
CIScreenBlendMode
CISepiaTone
CIShadedMaterial
CISharpenLuminance
CISixfoldReflectedTile
CISixfoldRotatedTile
CISoftLightBlendMode
CISourceAtopCompositing
CISourceInCompositing
CISourceOutCompositing
CISourceOverCompositing
CISpotColor
CISpotLight
CIStarShineGenerator
CIStripesGenerator
CISunbeamsGenerator
CISwipeTransition
CITorusLensDistortion
CITriangleTile
CITwelvefoldReflectedTile
CITwirlDistortion
CIUnsharpMask
CIVortexDistortion
CIWhitePointAdjust
CIZoomBlur

yuvconvolve

yuvconvolve.c

Convolution matrix for YUV streams

Performs a generic convolution filter on the video. Quite slow. Support any odd dimension matrix (3x3, 5x5, 7x7...) uses the command line argument -m 1,2,3,4,5,6,7,8,9.

I am thinking about adding support for predefined matricies, such as blur, sharpen, edge detection, emboss.

I now know that it's not wise to run sharpening on low bitrate mpeg files, as it highlights the macroblocks.

RESULTS


HISTORY

yuvcrop

yuvcrop.c

Crop, Matte and Matte detection

Used to detect matting in yuv sources. It can also be used to crop and matte a yuv video stream.

Cropping makes the destination video frame dimensions smaller, while matting will keep the video the same size, but replace pixels with a solid colour.

RESULTS


yuvdeinterlace

yuvdeinterlace.c

Double frame rate de-interlacer

A non destructive deinterlacer. Converts to a double frame rate, progressive yuv stream for further processing by temporal based non interlace aware filters. Then re-interlaced or frame rate converted before encoding.

This program uses an experimental detection algorithm to detect interlaced pixels and then uses cubic interpolation to replace pixels that are suffering comb effect. Image data is not lost as the frame rate is doubled. This adaptive algorithm is quite effective, however not ideal. Noisey artefacts may be apparent in (some) most material.

I have been looking into implimenting a 4 point DFT on the data and looking at the amplitude of the high (/2) frequency component. This has lead to better results but appears to still have issues at the boundary of interlace and non interlace material.

The above code could be extended to an arbitary number of pixels, comparing the /2 (interlace) frequency with the /4 frequency.

This release of the code I have played with comparing both the high frequency and the next frequency component. The high frequency shows areas of interlace however also detects edges. The second frequency component detects edges. By balancing these two results, by trial and error, I beleive to have come up with a better detection algorithm. However non interpolated interlace artefacts are still apparent. This is the best results so far.

Just when I thought I tuned the DFT algorithm to the best parameters. I came up with another method that simply looked for a "greater than, smaller than, greater than" pattern and as a result are now detecting interlace much more accurately than before. I also added an additional test on the AC value to eliminate non noticable interlace (or noise). This version is now available.

I am extremely happy with the results. Although not perfect, there are false positives, and a few false negatives. Of course I will be fine tuning this filter even more over time. For now I would use it as part of a yuv filter chain for my production videos.

TODO

Write a pixel merging algorithm for conversion of 25i material to 25p (rather than 50p).
Look at ways to perform anti-aliasing for fixing up nearest neighbour de-interlace filters. These sorts of filters should be shot, as it's quite easy to write a linear interpolator, not to difficult to write a cubic interpolator, and look I just wrote an adaptive type de-interlacer. I can't believe I saw this kind of de-interlace filter used on an episode of Top Gear the other night. Disgraceful!
Provide a screenshot...

Ideas

I have been experimenting with training a neural net to *learn* what the missing pixel may be from the surrounding pixels. My first attempts have not worked, if it is trained from images with many uphill lines, then uphill interpolation looks fantastic but downhill is very jaggered.

I am thinking about increasing the number of hidden layers in the neural net to see if it improves interpolation. The code is in java, I'll just need some encouragement to look at it again.

This part of the code would replace the cubic interpolator. Detection would still be separate algorithm, however comparing the interpolated pixel with the real pixel and replacing the real pixel with the interpolated if it is different by a threshold amount. May prove a better algorithm.

Example

by default yuvdeinterlace will double the frame rate and convert fields into full height frames. No arguments needed.

yuvdiag

yuvdiag.c

Video Diagnostics

Added new operating mode. Timecode burn. Will add a timecode stamp to the video. Also remember to download the font.

This tool has 4 operating modes. YUV split, which copies the U and V channels into the luma channel. Chroma Scope, which shows a histogram of the chroma values in intensity and Luma Scope, which shows a histogram of luminance on the Y axis. And a traditional histogram.

This program needs some work to add labels to the histogram scopes, but not being a video engineer, I do not know exactly what to add.

feedback would be appreciated.

RESULTS


yuvdiff

yuvdiff.c

Frame rate conversion detection

Started life similar to yuvaddetect, may even contain disused code from ad detect. This program produces an ASCII file suitable for plotting in gnuplot of the difference between consecutive frames or fields. Suitable for detecting 3-2 pulldown, or any other pulldown (frame duplication) frame rate conversion. If the output is ran through an FFT it can even detect other frequency rate changes such as 2.39 (25 to 59.94, progressive PAL to interlaced NTSC)

This program cannot remove pulldown. Try my other program yuvrfps for frame duplication removal. I do not know of a way to remove frame blended frame rate conversion.

I have used this extensively to detect pulldown material.

yuvfade

yuvfade.c

Black Fade

Will fade the video to black after X number of frames. Uses a trial and error method of fading to black, could be better.

I used this program to give a more professional feel to the downloaded internet videos I was converting

yuvGENERIC

yuvGENERIC.c

Nothing to see here move along

yuvilace

yuvilace.c

Interlace Detection

Experimental

Attempts to perform interlace detection, by FFT the vertical lines in the screen, and averaging the resulting frequency spectra. Also tried experimenting with filtering to intelligently remove interlace. I have no idea what state this code is in at the moment.

yuvrfps

yuvrfps.c

Frame Dropping Frame Rate Converter

Performs frame rate reduction by simple fractional amounts, by dropping 1 out of every X frames or fields. Can be used to convert 29.97 to 23.976 similar to yuvkineco. Or even 30 to 25 fps. It analyses X frames and finds the least different fram e and drops it, in theory removing any duplicated frames.

yuvwater

yuvwater.c

Flat Transparent watermark detection and removal

Attempts to detect and remove semi-transperant watermarks from the source. Produces a PGM file of the detected watermark which is used to remove or reduce the effect. This is a two pass process, the documentation is a little sparse.

The first pass produces a grey image file (PGM format) this is done by averaging luma of all the frames. The idea is that the watermark is consistantly brighter than the surrounding video. The PGM file may need to be edited (blurred) in places as other long term artifacts may be visible.

The second pass modifies the luma of the video reducing the brightness by the amount in the pgm file. This is not a linear reduction. Trial and error was used to determine the correct formula to reduce the luma by. For flat watermarks, this function is ideal.

Non flat watermarks cannot be removed by this program. Colour watermarks cannot be removed by this program. Solid watermarks cannot be removed by this program.

EXAMPLE

Pass 1: | yuvwater -d > watermark.pgm

May also use a black frame with the watermark showing, extracted by other means.

pass 2 | yuvwater yuvwater -m 145 -l 72 -u 384 -i watermark.pgm | -m specifies the amount to remove, the lower the number the darker the resulting watermark, good starting values are 140. -l specifies the black level for normalisation, good starting values between 32-80. If the black is too light, increase this value. -u specifies the white level, good starting value 384, if the white is too dark, decrease this value.

RESULTS


HISTORY

Isn't Chyler Leigh a goddess?