The problem:

When drawing inside polygons using a GPU shader, if your shader outputs areas containing high contrast sloping lines, those lines will be ailiased. Multisampling does nothing to fix the aliasing, as multisampling only serves to smooth the edges of geometry and not their interior fills.


Fig 1. Shading aliasing and smoothing

What is needed:

A methodology to perform anti-aliasing inside your shader.

My solution:

Determine the arc width θ of your smoothing distance, for example the arc across a pixel by diving your field of view by the number of vertical pixels in your scene.

Convert θ to foreshorten ratio ω using the equation ω = 1 / sin(θ). If your scene's resolution and aspect ratio won't change, this value ω may be held in a constant.

Account for the distance and slope of the surface by multipling ω, z depth, and then dividing by dot product of the eye vector to surface normal vectors to form Ω i.e. Ω = ω * z / dot(eye, normal).

Ω is now the of number world units you'll need to blend with when anti-aliasing your lines.

Fig 2. The example scene with both aliased and smoothed lines
Example, if we were to create a shader to draw concentric "bullseye" circles:

// rings are space 1 unit apart
let spacing = 1

// alternating red and white rings are sized equally
let line = spacing / 2

// segment is the position of our current fragment inside a ring
let segment = mod(distance(fragment.xy, origin), spacing)

// we are going to aa only the right side
if segment < line then
  color = red

// here is the aa part
else if segment < line + Ω then
  color = mix(red, white, (segment - line) / Ω)

// and we are done
else
  color = white

Fig 3. Aliased lines on the left[1], smoothed lines on the right[2]