Adding tiled backgrounds to UWP apps with Win2D

One of the many features that Silverlight never got from WPF was the ability to tile or repeat the image used on an ImageBrush. However, it had some workarounds like applying a custom pixel shader to get the same result – but, since the Silverlight version used on Windows Phone was even more limited, there hasn’t been a proper replacement (other than hacks based on displaying multiple images on a custom Panel control) in either Silverlight for Windows Phone, Windows Runtime or the new UWP platform.

Thankfully, now we have Win2D which combines the powerful graphical capabilities of Direct2D with the easiness of use of the XAML stack – and in this tutorial we are going to learn how to add a tiled background to our UWP app with very little code.

Approach

The premise is very simple – we will put an instance of Win2D’s CanvasControl in the root container of the page so it draws behind everything else, set both of its alignments to Stretch and fill it entirely using a CanvasImageBrush. Also, in the demo project, we will add some UI controls to tweak the scaling and opacity values to better showcase the flexibility of image brushes.

So, let’s get started!

Implementation

Start by adding the Win2D library from NuGet on your project, and as stated before, adding a CanvasControl element to the XAML page you want to. Hook up its CreateResources and Draw events and let’s jump to the code behind.

Declare two fields of type CanvasBitmap and CanvasImageBrush, as we will need them to hold the references to the background image and the brush that will be used for drawing, respectively. Then we can proceed to initialize them in the event handler for CreateResources that we added previosly – but wait, there’s a catch!

Win2D’s CanvasControl is nice enough to start drawing only when all resources have been loaded. But we have a problem: our event handler is of type void and we will need to do some asynchronous calls inside it; adding the async modifier to it won’t solve anything since async void methods are fire-and-forget. The answer is to use CanvasCreateResourcesEventArgs.TrackAsyncAction.

By wrapping all our async calls inside a Task and casting it to an IAsyncAction by calling AsAsyncAction, we can pass all our resource loading operation to the CanvasControl so it can track when they have completed, and start issuing draw events. Sounds a bit confusing? It can be at first, so check the following code snippet:

[code lang=”csharp”]
private void BackgroundCanvas_CreateResources(CanvasControl sender, CanvasCreateResourcesEventArgs args)
{
args.TrackAsyncAction(Task.Run(async () =>
{
// Load the background image and create an image brush from it
this.backgroundImage = await CanvasBitmap.LoadAsync(sender, new Uri("ms-appx:///Background.jpg"));
this.backgroundBrush = new CanvasImageBrush(sender, this.backgroundImage);

// Set the brush’s edge behaviour to wrap, so the image repeats if the drawn region is too big
this.backgroundBrush.ExtendX = this.backgroundBrush.ExtendY = CanvasEdgeBehavior.Wrap;

this.resourcesLoaded = true;
}).AsAsyncAction());

}
[/code]

As you can see, we just load our background image from disk and create the CanvasImageBrush with it. And to make it tile properly, we change both ExtendX and ExtendY to CanvasEdgeBehavior.Wrap – the default value is CanvasEdgeBehavior.Clamp which just extends indefinitely the last row or column of pixels.

That’s it for the initialization. Now let’s proceed with the drawing function, which is even easier, since we just fill a rectangle with the same size as the bound’s controls using the brush we have just created:

[code lang=”csharp”]
private void BackgroundCanvas_Draw(CanvasControl sender, CanvasDrawEventArgs args)
{
// Just fill a rectangle with our tiling image brush, covering the entire bounds of the canvas control
var session = args.DrawingSession;
session.FillRectangle(new Rect(new Point(), sender.RenderSize), this.backgroundBrush);
}
[/code]

Tiled background using Win2D

And that’s it! We now have a nicely tiling background. And to add a bit more of flexibility, we are going to explore the possibility of changing the size of the tiled image (by default it draws it with the same physical size as the source image) and its opacity. Add two Slider controls and hook their ValueChanged events – one for the scale, another for the opacity.

Let’s start with the opacity one. We will just divide its value by 100 (since Slider controls only support integer values, and the opacity range goes from 0.0 to 1.0) and assign it to the Opacity value of our ImageBrush. The call to CanvasControl.Invalidate will force a redraw of the background canvas so it updates accordingly:

[code lang=”csharp”]
private void OpacitySlider_ValueChanged(object sender, RangeBaseValueChangedEventArgs e)
{
// Don’t modify the brush properties if it hasn’t been initialized yet
if (!this.resourcesLoaded)
{
return;
}

// Change the opacity of the brush
this.backgroundBrush.Opacity = (float)(e.NewValue / 100.0);
this.BackgroundCanvas.Invalidate();
}
[/code]

Wait a minute – notice the resourcesLoaded flag? We added it so we don’t change any brush properties before it has been created. Just create it as a boolean field and set it to true at the end of the Task that creates your graphics resources.

And finally, let’s tackle the image scaling. This will be as easy as creating a Matrix3x2 that holds a transform (in our case, one that only has scaling) and assigning it to the brush’s Transform property. We can do lots of thing with this – rotate the brush image, skew it… your imagination is the limit!

[code lang=”csharp”]
private void ScaleSlider_ValueChanged(object sender, RangeBaseValueChangedEventArgs e)
{
// Don’t modify the brush properties if it hasn’t been initialized yet
if (!this.resourcesLoaded)
{
return;
}

// Apply a scale matrix transform to the brush; this way we can control how big the image will be drawn
this.backgroundBrush.Transform = System.Numerics.Matrix3x2.CreateScale((float)(e.NewValue / 100.0));
this.BackgroundCanvas.Invalidate();
}[/code]

Tiled background with opacity and scale selection controls

Source code

Get the full source code for this tutorial’s project on the Win2D-Samples GitHub repository – it’s sitting on the TiledBackground folder.

Creating custom video effects in UWP apps

One of the biggest shortcomings of WinRT for Windows/Windows Phone 8.1 was the state of media APIs – you could play a video file in a MediaElement but anything that involved extracting frames or applying effects was very cumbersome, as you had to put your code in a Windows Runtime Component, and worse of all, you had to write it in C++/CX.

Thankfully UWP has come to save the day, and while you still need to use Windows Runtime Components for writing this functionality (with its associated limitations, like making all your public classes sealed), now you can use C# for things like custom video effects, and more importantly, you don’t have to tweak the application manifest anymore to declare the Extension/ActivatableClass for them to be available! Other improvements include new classes like MediaClip, which allows you to load individual videos, and MediaComposition, which lets you create a media timeline to stitch together video clips and render them as a full length video.

In this tutorial we are going to learn how to create a basic custom video effect that implements the IBasicVideoEffect interface for rendering a video in grayscale, and how to apply it to a MediaClip before playing it through a MediaElement.

Implementation

We are going to create a new UWP application project and add a MediaElement to the main page’s XAML. We will come back to this later when we have our video effect ready – but you can try fiddling around and make it play the video on your own, it should look something like this:

Playing a video file in UWP

Now, add a new Windows Runtime Component project to the solution and add a new class to it. This is were we will implement our video effect based on IBasicVideoEffect, and this is very important as trying to implement it on another project type will throw a COMException with error messages Failed to activate video effect/Class not registered when trying to use the effect later.

Now that you have the class that will hold the code and it implements the appropriate interface, let’s walk through all the properties and methods that IBasicVideoEffect provides:

[code lang=”csharp”]
public bool IsReadOnly
{
get { return true; }
}
[/code]

IsReadOnly will return true if our effect doesn’t modify the input frame in any way. Since we are just going to get a source frame and apply a colour transform to it for drawing, we are going to set it as true.

[code lang=”csharp”]
public IReadOnlyList<VideoEncodingProperties> SupportedEncodingProperties
{
get { return new List<VideoEncodingProperties>(); }
}
[/code]

SupportedEncodingProperties returns a list of the video formats that our effect will support when outputting the processed video frame. As per the documentation, returning an empty list will give us frames in plain RGB format, which will be the easiest one to work on, so let be it.

[code lang=”csharp”]
public MediaMemoryTypes SupportedMemoryTypes
{
get { return MediaMemoryTypes.Cpu; }
}
[/code]

SupportedMemoryTypes is an enum that lets us specify if our effect will use the CPU, GPU or both for processing the frames. We want to set it to Cpu since this will give us access to the frame data through SoftwareBitmap, otherwise we would get handles to Direct3D data types.

[code lang=”csharp”]
public bool TimeIndependent
{
get { return true; }
}
[/code]

TimeIndependent lets us specify if our effect can be used while the playing video is in any playback state. We are going to modify the data per frame so we are going to set it to true.

[code lang=”csharp”]
public void Close(MediaEffectClosedReason reason)
{
}

public void DiscardQueuedFrames()
{
}

public void SetEncodingProperties(VideoEncodingProperties encodingProperties, IDirect3DDevice device)
{
}

public void SetProperties(IPropertySet configuration)
{
}
[/code]

These functions have pretty self explanatory names and we aren’t going to go through them since they aren’t used in this tutorial – however you can check the source code of the sample project for a brief explanation on what they are used for.

Finally, let’s explain what the ProcessFrame function does – the most important one, and the one we will be using for processing the frame data and applying our desired effect to the video clip.

[code lang=”csharp”]
public void ProcessFrame(ProcessVideoFrameContext context)
{
var inputFrameBitmap = context.InputFrame.SoftwareBitmap;

// Create intermediate buffer for holding the frame pixel data.
var frameSize = inputFrameBitmap.PixelWidth * inputFrameBitmap.PixelHeight * 4;
var frameBuffer = new Buffer((uint)frameSize);

// Copy bitmap data from the input frame.
inputFrameBitmap.CopyToBuffer(frameBuffer);

// Iterate through all pixels in the frame.
var framePixels = frameBuffer.ToArray();
for (int i = 0; i < frameSize; i += 4)
{
// Calculate the luminance based on the RGB values – this way we can convert it to grayscale.
var bValue = framePixels[i];
var gValue = framePixels[i + 1];
var rValue = framePixels[i + 2];

var luminance = ((rValue / 255.0f) * 0.2126f) +
((gValue / 255.0f) * 0.7152f) +
((bValue / 255.0f) * 0.0722f);

// Set the pixel data to the calculated grayscale values.
framePixels[i] = framePixels[i + 1] = framePixels[i + 2] = (byte)(luminance * 255.0f);
}

// Copy the modified frame data to the output frame.
context.OutputFrame.SoftwareBitmap.CopyFromBuffer(framePixels.AsBuffer());
}
[/code]

This function receives a ProcessVideoFrameContext as its only parameters, and it stores everything we need for applying our effect. The original frame for the video is stored inside the InputFrame member, and since we specified CPU processing, all the data for the frame image is in a convenient SoftwareBitmap in plain RGB (though in this case it’s ordered in BGR format) data.

We will start by getting the pixel data in a Buffer, and temporarily casting all its contents to a byte[]. Then, we iterate through it in increments of 4 bytes (since the frame has an alpha channel, too) and obtain the grayscale value via the relative luminance formula. Once this is done, we overwrite the R, G and B bytes with the luminance value, effectively obtaining a grayscale value with the same intensity as the source color. Finally, we copy our data buffer to the SoftwareBitmap of the output frame, and we will have a fully working custom video effect.That was easy, wasn’t it?

Back to our XAML page, hook up the Loaded event of the MediaElement (we have called it VideoPlayer since we will have to reference it later). Let’s open the code behind file and write the code for loading and playing the video file:

[code lang=”csharp”]
private async void VideoPlayer_Loaded(object sender, RoutedEventArgs e)
{
// Load video file
var videoFile = await Package.Current.InstalledLocation.GetFileAsync("big_buck_bunny.mp4");

// Create a MediaClip from the video file and apply our video effect
MediaClip clip = await MediaClip.CreateFromFileAsync(videoFile);
clip.VideoEffectDefinitions.Add(new VideoEffectDefinition(typeof(GrayscaleVideoEffect).FullName));

// Create a MediaComposition object that will allow us to generate a stream source
MediaComposition compositor = new MediaComposition();
compositor.Clips.Add(clip);

// Set the stream source to the MediaElement control
this.VideoPlayer.SetMediaStreamSource(compositor.GenerateMediaStreamSource());
}
[/code]

Pretty simple, isn’t it? We obtain a reference to the video file packaged with the app and create a MediaClip from it. Then, it’s a matter of adding a new VideoEffectDefinition to it with the full name (including namespace) of our custom video effect – the runtime will take care of fetching the type, activating and instancing it. Really easy compared to how convoluted it was in WinRT 8.1!

For displaying the video on the MediaElement, create a MediaComposition object and add the clip to it. Then it’s a matter of creating a MediaStreamSource from it with the GenerateMediaStreamSource function and passing it to our video playing control. The final result should look like this:

Grayscale video effect in action

While easy to do and good looking, this approach comes with some caveats. Since we are doing all our processing on the CPU side, computational load is big (it can quickly eat up one entire CPU core even with 720p videos) and memory usage is high since we are allocating a new buffer for each frame – this can be up to 60 allocations per second! It a future article we will look into doing it with GPU acceleration through the Win2D library.

Source code

The source code for this sample can be downloaded from the GitHub UWP-Samples repository, under the CustomVideoEffect subdirectory.

Invalid Pointer exceptions when using Win2D

Chances are if you are using Win2D in a complex app, you may have seen it crash sometimes with an unhandled exception with HResult 0x80004003 and a not very descriptive message containing only the phrase Invalid pointer.

If this is the case, take a second look at all your CanvasCreateResourcesEventArgs.TrackAsyncAction calls – because, while you may think this is a threading problem, the cause is far easier to solve: you are accessing a null object/property instead. Somehow Win2D is throwing an Exception with that error message instead of a NullReferenceException.