This is the seventh in a series of articles about OpenCL Fuse development, following the guidance of Vivo & Lowe’s Book of Shaders (BoS). In the previous article, I learned how to transform shapes by manipulating the coordinate system. This article expands on that approach by tiling the coordinate space.
I am going to get a bit adventurous this time by adding some image inputs to the Fuse that can be used to define patterns with exterior images. I’ll also look into passing information from one kernel to another, allowing me to use any of the previous modes as a source for tiles.
Before I get started here, I should mention that some of what I’m going here, specifically where it concerns handling external rasters, would probably be more efficient to just do in Lua. The reason I am doing this, though, is to learn about programming for GPUs, so don’t take what I do here as the “right” way to do it. I may eventually develop a Pattern Generator fuse in Lua for release.
Dynamic Input Creation
Up ’til now, none of my Fuses have been able to take an image input. All of them have generated their pixels completely internally. Now I need to not only add image inputs, but I want the Fuse to be able to take an arbitrary number of them. For an example of how this might work, I turn to the Switch Fuse, available on We Suck Less, courtesy of Stefan Ihringer. Switch has some interesting things going on. Each time you connect an input, it creates a new one. This works much like the Merge3D tool.
Switch is a very short Fuse—if you strip out the license and the bitmap icon, it’s only about 100 lines long. Nevertheless, it does some very cool stuff. And it’s simple enough that I can largely just copy-and-paste into my own Fuse! First, the Image Input:
InInput1 = self:AddInput("Input1", "Input1", {
LINKID_DataType = "Image",
LINK_Main = 1,
INP_Required = false,
INP_SendRequest = false, -- don't send a request for this branch before we actually need it.
})
Pretty straightforward. LINK_Main establishes the precedence order of the Input, with 1 being the gold Background input. If INP_Required is “true” then the Fuse will fail if there is no image available.
Next, a function that executes whenever something connects to that input:
-- OnConnected gets called whenever a connection is made to the inputs. A new
-- input is created if something has been connected to the highest input.
function OnConnected(inp, old, new)
local inpNr = tonumber(string.match(inp:GetAttr("LINKS_Name"), "Input(%d+)"))
local maxNr = tonumber(InWhich:GetAttr("INP_MaxAllowed"))
if inpNr then
if inpNr >= maxNr and maxNr < 64 and new ~= nil then
InWhich:SetAttrs({INP_MaxScale = inpNr, INP_MaxAllowed = inpNr})
self:AddInput("Input"..(inpNr + 1), "Input"..(inpNr + 1), {
LINKID_DataType = "Image",
LINK_Main = (inpNr + 1),
INP_Required = false,
INP_SendRequest = false,
})
end
end
end
Rather than taking this one top-to-bottom, I think it’s easier to start with the first line, then skip to the innermost portion. When a new connection is made to an input, the variable inpNr gets the integer from the end of that input’s name.
self:AddInput… That method is usually found only in the Create() function. Of course, there’s no reason to assume that Create() is the only place that inputs could be made, but until I looked at this Fuse it had never occurred to me to try it anyplace else. The method also shows the use of dynamic naming for the new input. The double period “..” is a Lua concatenation operator. It is used to connect two strings together. The result is that the newly created input will have the name “Input#”, where # is the next integer above the just-connected input. LINK_Main gets that same number.
Moving out one level, we see reference to an input called “inWhich.” The Switch Fuse uses this input to determine which image is going to be sent to the output. I’m going to be doing some internal compositing with all available inputs, so I won’t need that particular capability. I will, however, need some way to keep track of the maximum number of inputs that have been created, so I’m going to keep inWhich, but give it IC_Visible = false so the user will never actually see it. In any case, the line InWhich:SetAttrs({INP_MaxScale = inpNr, INP_MaxAllowed = inpNr}) sets some parameters on that input equal to the number of the input that’s just been connected. This allows the Fuse to keep track of how many inputs have been in use.
Out one more level, to the if statement:
if inpNr >= maxNr and maxNr < 64 and new ~= nil then
Once again, inpNr is the numerical value of the input just connected. This line makes three checks: The first to see whether the input is a re-used old one or the newest one. So if you replace the connection to Input2 on a four-input Switch, the Fuse will say “2 is less than 4, so I’ll skip this part.” The second makes sure that the maximum number of possible connections hasn’t been exceeded. I’m not sure if 64 is a hard limit on the number of inputs a node can have or if Stefan just though “that ought to be enough.” Finally, it also checks to be sure the input actually has something connected to it. OnConnected() will execute if you disconnect an input, too.
And back out to the highest level, where now that we understand what maxNr is for and where INP_MaxAllowed gets set, it makes more sense to query it here. To get a better feel for the state of the Fuse at any given point, uncomment the print() statements scattered here and there. They make it clearer what happens and when.
The last bit I need is a modification to OnAddToFlow(). I have that function in my Fuse already, but all it does is initialize the OpenCL Manager. Now it will also recreate the needed inputs when a saved comp is loaded. Stefan’s comments describe it quite well, so I’ll leave it at that.
The Patterns Tab
Next up, I’m going to finally add a tab to my Fuse. This is done by giving an Input the parameter ICS_ControlPage. Every new input that appears in the code below an input with that parameter will go onto the page specified. I am again looking ahead to where I want this Fuse to end up, so I am going to try something I haven’t yet seen in any other Fuses, but which should work, based on what I currently know. Here is my Tiling control:
InTiles = {}
InTiles[1] = self:AddInput("Tiles1", "Tiles1", {
LINKID_DataType = "Number",
INPID_InputControl = "SliderControl",
INP_Default = 1,
INP_MinScale = 1,
INP_MaxScale = 64,
ICS_ControlPage = "Patterns",
})
By using a list to hold the new controls, I hope that I can create new ones as needed. That is, I can use OnConnected() to create new slider controls in addition to new image inputs. It seems to be working pretty well, but I haven’t yet figured out how to call NotifyChanged() to cause the control panel to refresh. Switching to another tab or tool then back again causes the new input to appear.
Unfortunately, when I got to the point of trying to query the new inputs in Process(), it seems that the list isn’t working as expected. The new inputs exist, and they’re named correctly, but the InTiles list did not actually have anything other than the first one in it. That was momentarily disappointing, but I took another look at Switch to see if I could figure out what to do about it.
inp = self:FindInput("Input"..which)
That line can get an input and assign it to a variable, even if the input was not correctly assigned a handle upon creation. A little bit more hacking gets me this:
local tiles = {}
layers = 1
inp = self:FindInput("Tiles"..layers)
while inp do
tiles[layers] = inp:GetValue(req).Value
layers = layers+1
inp = self:FindInput("Tiles"..layers)
end
This creates a new list to hold each tiles value and a layers variable to hold the number of incoming images. The Tiles control will determine how many times the pattern repeats. Each image will get its own control, and at the end they’ll be composited together, allowing very complex patterns to be built relatively easily. We’ll see how it goes when I actually get it working!
I may eventually add additional controls to the Patterns tab. Right off the bat, I think I’ll probably want to be able to separate x- and y- tiling. I might save those extra features for a standalone Fuse, as this functionality will probably be worth distributing as a new tool.
Tiling
Pattern generation is about tiling. Execution is relatively simple. The BoS example uses a fract() function and a scale to do the job. The coordinate system is scaled up (causing the objects in it to appear smaller), and repeated. The fract() is applied to the scaled coordinates, and everything in front of the decimal point is discarded, leaving only the fractional portion. OpenCL also has a fract() function, but for some reason I can’t get it to work. Fortunately, the documentation tells exactly how the function works, with this esoteric formula:
fmin(x - floor(x), 0x1.fffffep-1f);
That second part of the fmin() is evidently the largest possible number smaller than 1.0. I’m going to take the Khronos Group’s word on that one because it seems to work just fine. Here is my new function that tiles the coordinate space:
float2 tile(float2 st, float2 tiles)
{
st *= tiles;
float2 temp = floor(st);
st = fmin(st - temp, 0x1.fffffep-1f);
return st;
}
So now all I need to do is to feed the tiles[1] variable into my kernels and put in a call to tile()!
Image Handling
Now it’s time to think about those image inputs and how they could be put to use. As with the tiling controls, I’ll need to fetch the images with FindInput() since they don’t have handles. Since each of the images should be associated with one of the Tiles inputs, I will simply grab the images at the same time as I do the Tiles values:
local tiles = {}
local img = {}
local layers = 1
local inp = self:FindInput("Tiles"..layers)
local temp = self:FindInput("Input"..layers)
while inp do
tiles[layers] = inp:GetValue(req).Value
img[layers] = temp:GetSource(req.Time, req:GetFlags())
layers = layers+1
inp = self:FindInput("Tiles"..layers)
temp = self:FindInput("Input"..layers)
end
Before I even get into tiling these new images, I need to figure out the compositing. There is a handy FuseMerge Fuse available that does exactly what I need to accomplish, so I’ll just go ahead and steal the necessary code from there. If you can’t already tell, 90% of my approach to programming is finding some code that already does what I need and adapting it. Just make sure you don’t “borrow” something with too restrictive a license or you may find yourself unable to distribute what you’ve made!
I’m not going to worry about supporting multiple blend modes at this point—a simple Over operation will do me fine. FuseMerge, of course, assumes that it’s dealing with only two images, while I am going to perform my operation with as many as the user cares to connect. That means I’ll need a loop, and I need to decide the layering order. In keeping with typical Fusion usage, I’ll put the lowest-order inputs in the back. That also conveniently makes the loop simpler:
for i=1, layers-1 do
out:Merge(img[i], {
MO_ApplyMode = "Merge",
MO_ApplyOperator = "Over",
MO_XOffset = 0.5,
MO_YOffset = 0.5,
MO_XSize = 1.0,
MO_YSize = 1.0,
MO_Angle = 0.0,
MO_FgAddSub = 1.0,
MO_BgAddSub = 1.0,
MO_BurnIn = 0.0,
MO_FgAlphaGain = 1.0,
MO_Invert = 1,
MO_DoZ = 0,
})
end
With this alone I have a multi-layer Merge node, which might itself be of use. If I add Blend and Apply Mode controls, Nuke users will appreciate it, I’m sure.
Rather than adding another button to the interface to put the Fuse into Pattern Generation mode, I will detect when something is connected to the inputs and use that state to determine the behavior of the tool. One more if statement encloses the entire OpenCL section of the Fuse:
if img[1] == nil then
I could write a test to find out if any input is connected, but I’m feeling lazy today, so I’ll assume that if the first input isn’t connected, then the user wants normal functionality. As for the tiling, I don’t think there’s any need to create a new mode for that. As long as the Tiling controls stay at their default of 1.0, everything works just as it did before.
Way down at the bottom of the Fuse, I add an else clause with instructions for when Input1 is connected to something. First, I’ll need a temporary buffer to hold the output of the kernel. I think I’ll leave the Transform options available (though I’ll need to adjust the control panel again to expose them when they’re needed), so I’ll also copy in the angle conversion. And I’ll set up a new kernel:
else
out:Fill(p)
temp = {}
-- convert angles to radians and invert
angle = angle * math.pi/180 * -1
-- precalculate sin and cos
local sin_theta = math.sin(angle)
local cos_theta = math.cos(angle)
for i=1, layers-1 do
if img[i] then
temp[i] = Image(imgattrs)
local outcl = prog:CreateImage(initimg, "write")
local kernel = prog:CreateKernel("pattern")
if kernel then
prog:SetArg(kernel, 0, outcl)
prog:SetArgInt(kernel, 1, img.Width, img.Height)
prog:SetArg(kernel, 2, shapeWidth, shapeHeight)
prog:SetArg(kernel, 3, center.X, center.Y)
prog:SetArg(kernel, 4, sin_theta, cos_theta)
prog:SetArg(kernel, 5, tiles[1], tiles[1])
success = prog:RunKernel(kernel)
end
if success then
success = prog:Download(outcl, temp)
end
--Merge operation goes here
end
end
end
This code will call the pattern kernel for each connected input then stack the layers.
The Pattern Kernel
I am wandering rather far afield from BoS’s lessons. There is nothing there about manipulating incoming images, so I am going to have to dig around somewhere else to figure out how to do what I want. What I need to do is use all of my coordinate-changing functions to determine which pixel in my source image I want to sample for inclusion in my output image. It was a quite short hunt because the OpenCL sample Fuse described in the VFXPedia has exactly what I need: an API function FuSampleImageCf(). I replace all of the shape-generation code in my existing kernels with a single line:
float4 outcol = FuSampleImageCf(src, st, imgsize);
And that does exactly what I want it to do. Much to my surprise!
I recall having a hard time digging up that sample Fuse when I originally went looking for it. I think it was in a developer’s folder in Fusion 7.0 or something like that. For convenience, you can download it here:
Extended capabilities
After playing with my new toy for a little while, its limitations start to become apparent. For one thing, I haven’t given any thought to input images that aren’t the same size and aspect as the output. For another, it would be great if the tiles would wrap from one side to the other. It would also be cool to be able to create variations by tile in orientation, color, and more.
This Fuse looks useful enough, though, that I think I want to fork it to a stand-alone tool rather than continuing to develop it in this one. It may therefore be a little while before I continue this series. When the Pattern Generator is finished I’ll share it both here and at We Suck Less.
Here is the current incarnation of the Fuse: BoS_LessonNine.fuse
The next two chapters at BoS are about adding some chaos through randomness and noise. I don’t know if I’ll tackle all of that in a single article or two (or even three), but by the end I should have at least two new noise generators to supplement Fusion’s existing Fast Noise (which I believe is Simplex noise, if I recall correctly).
