Another feature I really wanted to implement in ND2D were masks. Just like the setMask() method in flash. In Stage3D (OpenGL), there is no such thing as a mask. You can display textured triangles, that’s it, but you know that nearly everything is possible with a pixel shader. So let’s start:
The idea of masking in a fragment shader is to grab the pixel color of your texture, then grab the pixel color of your mask, multiply the two colors and display the result. But how do we find the correct pixel in the mask? Our task is to find the right UV coordinates for the mask texture.
If you look at the above image, the mask is rotated and overlaps the sprite we want to mask. How do we find the correct pixel (UV coordinate) of the mask, that overlaps this orange pixel in the sprite? Somehow we have to map the position of the pixel in the sprite to the pixel in the mask and we can do that by transforming it between the different coordinate systems. In a vertex shader we calculate the final pixel positon from local space to world space. The idea is to map this pixel in world space back to the local coordinate system of the mask. This way it’s pretty easy to find the correct UV coordinates. Let’s do a simple actionscript test:
// this is the top right corner of our sprite quad.
var v:Vector3D = new Vector3D(128, -128, 0, 1);
// this is the sprites matrix, translated a bit
var clipSpaceMatrix:Matrix3D = new Matrix3D();
clipSpaceMatrix.appendTranslation(100, 0, 0);
// this is the masks matrix, it's in the same position as the sprite
var maskClipSpaceMatrix:Matrix3D = new Matrix3D();
maskClipSpaceMatrix.appendTranslation(100, 0, 0);
// this is the masks size
var maskBitmap:Rectangle = new Rectangle(0, 0, 256, 256);
// invert the matrix, because we want to map back from world space to local mask space
maskClipSpaceMatrix.invert();
// transform our vertex from local sprite space to world space
v = clipSpaceMatrix.transformVector(v);
[trace] moved to clipspace: Vector3D(228, -128, 0)
// transform world space vertex back to local mask space
// the result is the same vector of course, because the positions of mask and sprite are equal
v = maskClipSpaceMatrix.transformVector(v);
[trace] moved to local mask space: Vector3D(128, -128, 0)
// calculate the uv coordinates from the local pixel position
v = new Vector3D((v.x + (maskBitmap.width * 0.5)) / maskBitmap.width,
(v.y + (maskBitmap.height * 0.5)) / maskBitmap.height,
0.0, 1.0);
// the result is what we expect, the top right uv coordinate:
[trace] local mask uv: Vector3D(1, 0, 0)Porting this idea to a shader is pretty straight forward. Let’s code a PB3D Material Shader:
void evaluateVertex()
{
interpolatedUV = float4(uvCoord.x + uvOffset.x, uvCoord.y + uvOffset.y, 0.0, 0.0);
float4 worldSpacePos = float4(vertexPos.x, vertexPos.y, 0.0, 1.0) * objectToClipSpaceTransform;
// maskObjectToClipSpaceTransform is the invertex clipspace matrix of the mask
float4 localMaskSpacePos = worldSpacePos * maskObjectToClipSpaceTransform;
// halfMaskSize.xy is maskBitmap.width/height * 0.5 passed as a parameter
// invertedMaskSize.xy = 1.0 / maskBitmap.width/height passed as a parameter, because divisions are not properly working in the current pb3d release
interpolatedMaskUV = float4((localMaskSpacePos.x + halfMaskSize.x) * invertedMaskSize.x,
(localMaskSpacePos.y + halfMaskSize.y) * invertedMaskSize.y,
0.0, 0.0);
}
void evaluateFragment()
{
float4 texel = sample(textureImage, float2(interpolatedUV.x, interpolatedUV.y), PB3D_2D | PB3D_MIPNEAREST | PB3D_CLAMP);
float4 texel2 = sample(textureMaskImage, float2(interpolatedMaskUV.x, interpolatedMaskUV.y), PB3D_2D | PB3D_MIPNEAREST | PB3D_CLAMP);
result = float4(texel.r * color.r * texel2.r,
texel.g * color.g * texel2.g,
texel.b * color.b * texel2.b,
texel.a * color.a * texel2.a);
}If you don’t want to use PixelBender3D and like to ‘torture’ yourself with AGAL, you can write the same shader this way:
/*
vertex shader:
vc0-vc3 = clipspace matrix of sprite
vc4-vc7 = inverted clipspace matrix of mask
vc8.xy = half mask width / height
vc8.zw = mask width / height
va0 = vertex
va1 = uv
*/
m44 vt0, va0, vc0 // vertex * clipspace
m44 vt1, vt0, vc4 // clipspace to local pos in mask
add vt1.xy, vt1.xy, vc8.xy // add half masksize to local pos
div vt1.xy, vt1.xy, vc8.zw // local pos / masksize
mov v0, va1 // copy uv
mov v1, vt1 // copy mask uv
mov op, vt0 // output position
/*
fragment shader:
*/
mov ft0, v0 // get interpolated uv coords
tex ft1, ft0, fs0 <2d,clamp,linear,nomip> // sample texture
mov ft2, v1 // get interpolated uv coords for mask
tex ft3, ft2, fs1 <2d,clamp,linear,nomip> // sample mask
mul ft1, ft1, ft3 // mult mask color with tex color
mov oc, ft1 // output color
The result is visible here: ND2D – alpha masks (Move your mouse over the crates). I added one more feature: You can set the alpha of a mask, that means that you can specify how much the mask affects the sprite. In the demo above the alpha fades from 0.0 to 1.0. Since we’re using all four color components in our calculations (r,g,b,a), we can not only mask the alpha, but all color channels. I don’t know if this it’s a “nice thing to have” or if it will get annoying when you use sprites as masks in your game and need to provide an extra image for that. Just let me know :) Here is the example: ND2D – disco color masks.
