Geometry shaders for Dummies

DirectX 11 offers 2 new programmable shader stages in the graphics pipeline, the Hull Shader and the Domain Shader. Before I investigate those, I wanted to first look at Geometry shaders, which was actually introduced in DirectX 10 already. Geometry shaders allows you to treat your data at a primitive level rather than the vertex level. Where in a Vertex shader you have access to only a single vertex at a time, the geometry shaders gives you the three vertices of a triangle as input. The shader can then modify the vertices, add more triangles or even discard it entirely. For my example I will tessellate an Icosahedron. This is a convenient shape to subdivide since it results in a sphere with evenly distributed vertices. Each triangle of the original icosahedron is recursively subdivided into 4 triangles. The newly created vertices are then normalized so that the lie on the surface of a unit sphere.
The vertex shader which would usually transform the vertices form local space to screen space, will now just pass the vertices along to the pipeline without any transformation.

struct GSInput
{
 float4 Position    : POSITION;
};
GSInput VertShader(VSInput input )
{
 GSInput Out;
 Out.Position  = input.Position;
 return Out; 
}

Note that the original normal of the vertices are not really of interest since for a unit sphere, the normal is the same as the position. We only pass on the vertex position in local space.
SphereVertex is a helper function used by the Geometry shader that now does the vertex transformation.

PSInput SphereVertex(float4 position)
{
 PSInput output;
 output.Position = mul( position, WorldViewProjection );
 output.Normal = mul(position.xyz, (float3x3)World);
 return output;
}

Below is the algorithm for a single subdevision of an icosahedron. Here I ran into a limitation of HLSL. It turns out that recursive functions are not supported... One could write the algorithm iteratively but it is a bit more messy, and this exercise was really just a stab creating a geometry shader that does something useful. DirectX 11 actually has a hardware tesselator for this sort of thing. In the next tutorial I will create Hull and Domain shaders to do tessellation with arbitrary complexity.

[maxvertexcount(12)]
void GeomShader( triangle GSInput input[3], inout TriangleStream OutputStream )
{ 
  float4 va = input[0].Position;
  float4 vb = input[1].Position;
  float4 vc = input[2].Position;
  float4 v1 = float4(normalize(va.xyz + vb.xyz),1);
  float4 v2 = float4(normalize(va.xyz + vc.xyz),1);
  float4 v3 = float4(normalize(vb.xyz + vc.xyz),1);
  OutputStream.Append( SphereVertex(vc) );
  OutputStream.Append( SphereVertex(v2) );
  OutputStream.Append( SphereVertex(v3) );
  OutputStream.Append( SphereVertex(v1) );
  OutputStream.Append( SphereVertex(vb) );
  OutputStream.RestartStrip();
  OutputStream.Append( SphereVertex(va) );
  OutputStream.Append( SphereVertex(v1) );
  OutputStream.Append( SphereVertex(v2) );
  OutputStream.RestartStrip();
}

When setting up the geometry, one creates a geometry shader in very much the same way as a vertex or pixel shader.

type Geometry(device, shape) =
...
    let gsByteCode =    ShaderBytecode.CompileFromFile(
                            "Simple.hlsl", 
                            "GeomShader", 
                            "gs_4_0", 
                            ShaderFlags.None, 
                            EffectFlags.None)

    let geomShader = new GeometryShader(device, gsByteCode)
....
    member this.Prepare(transforms) =
....
        do deviceContext.GeometryShader.Set(geomShader)
        do deviceContext.GeometryShader.SetConstantBuffer(
            vsConstBuffer.Update(transforms),0)

Note that the transforms constants (World/WorldViewProjection) that were previously used by the vertex shader must now be provided to the geometry shader instead.

For interest sake, here is what the subdivision algorithm in F# looks like if it were to be done on the CPU instead.

    static member Sphere subdivisions =
        let t = (1.0f + sqrt(5.0f))/2.0f;
        let s = sqrt(1.0f + t*t);
        let icosahedronVertices = [|
            Vector3(    t, 1.0f, 0.0f)/s;
            Vector3(   -t, 1.0f, 0.0f)/s;
            Vector3(    t,-1.0f, 0.0f)/s;
            Vector3(   -t,-1.0f, 0.0f)/s;
            Vector3( 1.0f, 0.0f,    t)/s;
            Vector3( 1.0f, 0.0f,   -t)/s;    
            Vector3(-1.0f, 0.0f,    t)/s;
            Vector3(-1.0f, 0.0f,   -t)/s;
            Vector3( 0.0f,    t, 1.0f)/s;
            Vector3( 0.0f,   -t, 1.0f)/s;
            Vector3( 0.0f,    t,-1.0f)/s;
            Vector3( 0.0f,   -t,-1.0f)/s;
            |]
        
        let icosahedronIndices = [|
            0; 8; 4;
            1;10; 7;
            2; 9;11;
            7; 3; 1;
            0; 5;10;
            3; 9; 6;
            3;11; 9;
            8; 6; 4;
            2; 4; 9;
            3; 7;11;
            4; 2; 0;
            9; 4; 6;
            2;11; 5;
            0;10; 8;
            5; 0; 2;
           10; 5; 7;
            1; 6; 8;
            1; 8;10;
            6; 1; 3;
           11; 7; 5;
        |]

        let rec subdevide(vertices:array, indices, level) = 
            if level < subdivisions then
                let subdivideHelper(accVertices:array,
                                    accIndices,
                                    triangle:list) index =
                    match triangle with
                    | ib::ia::[] ->
                        let ic = index
                        let va, vb, vc = vertices.[ia], vertices.[ib], vertices.[ic]
                        let v1, v2, v3 = 
                            Vector3.Normalize(va + vb), 
                            Vector3.Normalize(va+vc), 
                            Vector3.Normalize(vb + vc)
                        let i1 = accVertices.Length
                        let i2 = i1+1
                        let i3 = i2+1
                        (Array.append accVertices [|v1;v2;v3|], 
                         Array.append accIndices  [|i1;i3;i2; 
                                                    ia;i1;i2;  
                                                    ic;i2;i3; 
                                                    ib;i3;i1|],
                         [])
                    | _ ->  (accVertices,accIndices, index::triangle)
                
                let vertices, indices, _ = 
                    Array.fold subdivideHelper (vertices,[||],[]) indices
                subdevide(vertices, indices, level+1)
            else
                vertices, indices

        let vertices, indices = subdevide(icosahedronVertices, icosahedronIndices, 0)
        { Vertices = Array.map (fun vec -> Vertex(vec, vec)) vertices; Indices = indices }