Colored Outline Shader in Unity

It seems like I’m on a roll with outline shaders! I’ve already written a basic cel shader with outline tutorial and an animated dotted outline shader, and now I’m doing this colored outline.

The special thing about the color of this outline is that it’s using the color of the texture, so that the outline color always matches the pixel it’s closest to. I find it’s a common tool used by 2D artists to make outlines feel more natural, as opposed to using a solid color like black for the whole outline.

Here’s an example of the difference it makes with an example from 2D pixel art from u/croakiee on Reddit:

outlineEx

This tutorial teaches you how to do this effect in 3D.

IMPORTANT: Since I already went over the basics of how to do the outline here, I’m not going to repeat that explanation in this tutorial. Do the basic tutorial first if you’re not sure how to create an outline at all.

Here’s the full code for the shader for reference, under a non-commercial open-source license:

->> Link to full code for Unity Colored Outline Shader

Also, The 3D model is from this free asset pack.

Now, on with the tutorial!


Colored Outline

This effect is quite simple, and happens completely inside the vertex shader.

If you remember from the basic outline tutorial, our outline is created by scaling the original model along each vertex’s normal direction. Now, to make the outline color the same as the vertex color, we simply sample the vertex color and add it to the vertex shader output.

Firstly, add a color to your vertex output struct:

struct vertexOutput {
    float4 pos : SV_POSITION;
    float4 color : TEXCOORD0;
}

And add one extra line of code in the vertex shader to sample the color:

output.color = tex2Dlod(_MainTex, float4(input.texCoord.xy, 0, 0));

And your output should look like this:

outlineColorstep1

Why sample the color at each vertex, you might ask, instead of at each fragment like we usually do? Basically, this is a lazy way of blurring the outline color. The vertex shader is run less often (only for every vertex on the mesh) than the fragment shader (basically for every pixel), and the colors between each fragment are interpolated, which essentially blurs away details. If we sample at every fragment instead, we get too much detail, and the outline ends up looking like this:

outlinecolorpixel
oh god, the eyes

Now that you’ve got your outline sampling the vertex color, let’s make the outline distinguishable from the model’s texture. As you can see right now, they’re a little bit too similar.

To do this, we’re going to add an outline color property and multiply our sampled color by the outline color

output.color = tex2Dlod(_MainTex, float4(input.texCoord.xy, 0, 0));
output.color *= _OutlineColor;

Tune the outline color to a dark grey to get this final result! You can, of course, use whatever color and blending mode you want.

coloredOutline

Good job, you made it!


Fin

Here’s the link to the full code for the Unity Colored Outline Shader again. The full code also includes a cel-shaded lighting pass and shadow pass- if you only want to look at the outline technique, look for the pass labled “// Outline pass”.

I hope you learned something from this tutorial- if not a new technique, possibly just some inspiration for how to expand on common techniques like 3D outlines 🙂

If y’all have any questions about writing shaders in Unity, I’m happy to share as much as I know. I’m not an expert, but I’m always willing to help other indie devs 🙂 And do give me feedback about the tutorials, I love hearing from y’all!

Good luck,

Lindsey Reid @so_good_lin

Published by

Linden Reid

Game developer and tutorial writer :D

2 thoughts on “Colored Outline Shader in Unity”

  1. Hey Linden, these outline shader tutorials are great! How would I go about modifying the outline to change color based on whether the pixel is in shadow? So a black outline turns white when in darkness? You already have a shader pass so it seems as the groundwork has already been laid. Any light you could shed would be helpful.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s