A game of Tricks II – Vertex Color

Vertex color is my friend. It should be yours too really… plenty of cool stuff to do. If you’re not familiar with it, let me introduce the concept. Vertex color, or vcolor, is essentially just a color with RGB and A channels stored for each vertex of a mesh. It’s a pretty standard, pretty old and classic 3D feature. Originally, the main use of vertex color was to allow color variations on large 3D surfaces with a single or limited number of textures. Back in the days, you’re typical video card couldn’t pile up to 8 or even 4 textures on the same triangle. Even blending two textures was considered expensive and mapping large expenses of earth or grass often ended in ugly tiling surfaces. But you already had the possibility to use vertex color to add variety to break the tiling effect and make richer visuals at a small cost. As a gamer, my most vivid memory of intense vertex color use is the first Planetside game (my first and last really enjoyable MMO experience).

vcolor_planetside2

On each of these pictures the ground is made of one and only one tiling texture. The earth roads whether, gray or brown, are using the exact same texture as the green grass. The roads and large patches of slightly different colors are defined by vertex color only. It’s more clear on the aerial view where you can distinguish vague triangular shapes. So, how does it work… Well, either the artist adds vertex color ‘by hand’ or the developer procedurally : one color for each vertex of the mesh. Then it’s up to the 3D material or more precisely to the shader to decide what to do of this color. The typical way of using this color is to multiply it with the diffuse texture color. That’s what was done in Planetside :  a light gray strands of grass texture multiplyed by a green color for the actual grass and a brown color for the roads.

vcolor_planetside1

Vertex color is cheap (maybe not so much on mobile though) because it’s a per-vertex information. That’s also its main drawback : the vertex color ‘resolution’, if I may say so, is limited by the actual tessellation of the object. With a dense mesh, you can have vertex color fitting closely to small details but with large polygons you may end up with only gradients which may betray the triangular nature of your 3D scene. 3D cards interpolate linearly all per-vertex informations on each pixel (or fragments to be more precise) of a triangle drawn on screen. It’s what happens also for normals or uv coordinates. Each fragment of a triangle gets a color mix of the 3 colors assigned to the 3 vertices with weights depending on the distance between the fragment and each vertex… Ok, let’s make it clearer : if you get closer to a vertex, you get more of its color and less from the 2 others.

vcolor_triangle

When you learn 3D modelling, you are often taught that you must keep using only quads. It means that your triangles must come in pairs of two and be coplanar (be on the same plane). Even if it can cost a fair amount of additional triangles, it has several advantages : it’s far more easy to tesselate or smooth the model further with a clean result and you avoid a lot of lighting problems associated with low poly models.  If you don’t use quads, you may end up with disgracious shading showing clearly the limits of your triangles and you’ll have to ‘turn edges’ all over the place to get things right. With vertex color, it is even worse : even if you’re using quads, it won’t help and you may have to ‘turn edges’ to get closer to what you want. Here is an example of a sparsely tessalated plane grid. On each plane only one vertex has been assigned with a blue color. On the left plane all quads are divided in the same direction, on the right plane i’ve turned two edges around the blue spot. See how the resulting blue shape is quite different.

vcolor_turnedge

By now you should have guessed, that you’ve got to use vertex color properly to get decent visual results. You must apply it carefuly and sometimes even plan your modelling specifically just to have the required support for your colors. For instance, if you want clean cuts along some edges and avoid having two adjacent faces with vertex colors mixing between them, you just have to ‘split’ vertices. Take a look at this plane grid. It’s only one object. Vertex color is mixing in the top and bottom rows but not in the middle.

vcolor_splitedge

To achieve this, you have to split the 3 highlighted vertices in 6 vertices (3 for the left polygons and 3 for the right). You have to do this because you can’t have two values of vertex color on a single vertex. Even if you could, how the video card could decide which color must be assigned to which triangle ? These 3 vertices either must have blue or red but can’t have both assigned. This is quite interesting because you can understand an important fact with this. Something that has nothing to do with vertex color but which is critical for performances. As I said earlier, vertex color is a per-vertex information like uv coordinates or normals. For one vertex, you can have one normal and one uv coordinate, no more (uv2 is a completely independent channel so really it’s like a third per-vertex information). When you map an object, if on 2 adjacent triangles you want to different parts of a texture with a gap between, it means that their 2 common vertices must point toward 2 regions of the map. This is impossible if you don’t split the vertices at the edge to get 4 vertices. The first set of two vertices will point toward one region of the map for one of the polygon and the second set will point to the other region. But no 3D package asks the user to cut the meshes when unwrapping an object, it’s done behind the scene, automatically. Some softwares don’t even have to comply with these constraints because they do not rely on hardware acceleration and can use more complex data structures for meshes. But whether your 3D production package is splitting vertices or not along seams of UVs, normals or vcolor, vertices will be split when you import the models in a real time engine like Unity. This operation may lead to a far greater number of vertices announced by Unity than what you were told by 3DSmax or Maya. Do not be surprised.

Let’s get back to the original subject of this post and talk a bit about how you actually add vertex color to a mesh. As usual for the artists, I can only talk about 3DSMax which is the only one I know fairly well. To add and edit vertex color, you have to add the ‘Vertex Paint’ modifier. This modifier comes with a panel of tools to show / hide and edit vertex color. You can paint it directly on objects and even create layers like in photoshop. Be aware though that you can export only the final results, not individual layers. If you try to render your scene, whether you have activated vcolor in the viewport or not, you won’t see it on your objects. Why ? Because you never told the material to make something of it ! Vertex color is an information stored in the mesh but it’s not necessarily displayed. If you want to preview the ‘classic’ multiplication of a diffuse texture with the vertex color, here is the set up you must create in the material editor :

vcolor_materialeditor

In the diffuse slot add an RGB multiply map with one bitmap and one vertex color map as children. If you apply this material to an object without a vertex paint modifier, the vertex color map will be useless. If you want to see the same result in Unity, you’ll have to create or find a shader that is actually using the vcolor information. More about that a bit later… One other potentially interesting thing in 3DSMax concerning vcolor is the Assign Vertex Colors tool (also included in the Vertex Paint modifier) : you can bake the content of your diffuse map or the lighting (including shadows and radiosity) or both in the vertex color. It means you can sample the color of your diffuse or of the lighting for each vertex and transfer the result in the vcolor channel. Baking projected shadows will require a very tessellated object if you want a decent result which may be less optimal than using a classic lightmap but in a very few special cases it may be interesting. Once the automatic process has been applied you can still paint manually over the result to correct imperfections. Of course, you don’t have to necessarily multiply your vertex color with the diffuse component. You can do whatever you want : add, substract, use it as a mask… You can use it to apply tint variations or shading in place of a lightmap for simple geometry. One small trick : if you want to make the most of vcolor when you’re multiplying it with a texture, you might want to multiply the result by 2 (or more). Why is that ? Multiplication is fine and cheap but it’s limited : you can only darken things. Multiplying by white (1,1,1) does nothing and everything else is darkening the result. If you multiply a texture by vcolor then by 2, every color lighter than the average gray (0.5/0.5/0.5) is lightening the result and every color darker than this gray is darkening the result. Even if you have half the precision, you have more control. You could also decide to add or substract whether you’re above or under the average gray for other effects…

In the picture below (click to enlarge), you’ll see a very simple low poly scene and the texture atlas used for the entire scene. You can guess how vertex color has been used to tint the leaves and bushes in green, darken and lighten parts of the roofs and create fake simplistic ambient occlusion (ie shadows created by corners and overhanging roofs)

village_low_poly_thumb

Let’s talk a bit about the coding part in Unity. My goal here is not to show freakingly complex shader stuff. I’m no shader guru but I’ll show you one or two things to use vertex color in useful or unexpected ways. Here is one of the most simple shader to take advantage of vertex color :

Shader "LitVcolor" {
    Properties {
       _MainTex ("Base (RGB)", 2D) = "white" {}
    }
    SubShader {
       Pass {
           Lighting On
           ColorMaterial AmbientAndDiffuse
           SetTexture [_MainTex] {
              combine texture * primary DOUBLE
           }
       }
    }
}

It’s a fixed function shader, the simplest form of shader you can use in Unity. ‘ColorMaterial AmbientDiffuse’ is where you actually turn on the vertex color magic. Because of it, in the following block, the primary term represents the vertex color value. Note how primary is followed by ‘DOUBLE‘ which is doing nothing more than multiplying the value by two. If you apply this shader to an object with no vertex color assigned, it will effectively multiply all diffuse texture components by two because vertex color is white by default. If you want the vertex color to have no effect, you have to apply an average gray (0.5/0.5/0.5) to all vertices, manually in a 3DSMax or by script. Let’s take a look at a quick example of how you actually manipulate vertex color by script to do just that!

// retrieve a reference to the Mesh of the object from the 
// MeshFilter component
Mesh mesh = myGameObject.GetComponent<MeshFilter>().mesh;

// get the vertex count for the objet
int vCount = mesh.vertexCount ;

// create an array of Colors with the same length
Color[] new_vcolor = new Color[vCount];
for (int i = 0 ; i < vCount ; ++i)
{
   // set the color for the corresponding vertex index in the array
   // we choose a gray color with an alpha of 1
   new_vcolor[i] = new Color (0.5f,0.5f,0.5f,1) ; 
}

// replace the current vertex color array 
//(white everywhere if it was untouched) 
// with the newly created one
mesh.colors = new_vcolor ;

Please note that like any data set from the Mesh class (vertices, colors, uv, uv2, normals, etc…), you can’t modify the value of individual vertices like this :

for (int i = 0 ; i < vCount ; ++i)
{
    mesh.colors[i] = new Color (0.5f,0.5f,0.5f,1) ;
}

You can retrieve a copy of the whole array or set a new array as a whole. If you want to partially modify one of the data set. You have to retrieve a copy, modify this copy and reinject it in the mesh. This is one of the first gotchas of procedural mesh manipulations but I’ll get back to this in a later post.

Ok so now let’s move a notch up the complexity of shaders. Here is a slightly simplified shader from Transcripted. It’s a surface shader : a Unity specific class of shaders. Basically it looks like a CG shader but a lot of the hassle of basic features is handled for you making it a lot shorter than your usual full CG shader. It has also the advantage of taking car for you of the platforms specific subtleties if you intend to compile a cross platform project.

Shader "defaultSpriteShader" {

 Properties {
 _Color ("Main Color", Color) = (1,1,1,1)
 _SpecColor ("Specular Color", Color) = (0.5, 0.5, 0.5, 1)
 _Shininess ("Shininess", Range (0.01, 1)) = 0.078125
 _MainTex ("Base (RGB) Gloss (A)", 2D) = "white" {}
 _BumpMap ("Normalmap", 2D) = "bump" {}
 }

 SubShader {
 Tags {"Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent"}
 LOD 700
 CGPROGRAM
 #pragma surface surf BlinnPhong alpha
// surface = it's a surface shader
// BlinnPhong = a type of built-in lighting formula
// alpha = this shader use transparency
// surf = the name of the fragment surface shader defined later

 sampler2D _MainTex;
 sampler2D _BumpMap;
 float4 _Color;
 float _Shininess;

 // definition of the input structure sent to the fragment surface
 // shader for each fragment/pixel
 struct Input {
     float2 uv_MainTex;
     float2 uv2_BumpMap;
     // here we tell Unity that we intend to use vertex color 
     // in the fragment shader
     float4 color : COLOR;
 };

 // the fragment surface shader definition
 void surf (Input IN, inout SurfaceOutput o) 
 {
     // the base color of each pixel is defined
     // by the main texture
     half4 tex = tex2D(_MainTex, IN.uv_MainTex);

     // _Color is the color defined in the material inspector
     // it's not the vertex color!
     half4 baseColor = tex * _Color ;
     half4 c = tex * _Color;

     // IN.color is the vertex color value interpolated for this
     // particular fragment/pixel.
     // Here, we choose once more to multiply the albedo(=diffuse)
     // value of this pixel by the vertex color and then by 2
     o.Albedo = c.rgb * IN.color.rgb * 2;

     // the final alpha of each pixel is also multiplied by
     // the vertex alpha
     o.Alpha = c.a * IN.color.a; 

     // Here we add normal mapping and specular effect
     // that's why surface shaders are cool, quite simple...
     o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv2_BumpMap));
     o.Specular = _Shininess;
     o.Gloss = 1;
 }
 ENDCG
 }
}

This shader is a lot more complex and expensive than our previous fixed function example but that is the price to pay if you want normal mapping, specular effect and stuff like that!

One of the most important line here is :

o.Alpha = c.a * IN.color.a;

Why is that? Well, if you try to learn about performance optimizations in 3D, you’ll soon hear everyone talking about the famous ‘draw calls’. Modern 3D cards are good at drawing vast amounts of triangles but they perform much better if all these triangles share a common material. Each time, they must handle the drawing or rasterization of a new object and even more if the new object use a different material from the previous one, there’s a price to pay in term of performances because of these draw calls. 3D engines like Unity have the capacity of batching or merging object with the same material to decrease this cost without you doing anything. You get bigger objects but less of them, hence less draw calls which leads to better performances. But to do that, you must actually use the same material. The very exact same material : same shader, same textures, same input values.

When you create a 2D games with sprites, it’s likely you’ll use very often the same shader. If you want to use the same texture, you’ve got to use atlases or sprite sheets. That’s the base of every single 2D framework done in Unity. It’s quite easy to do and I’ll tell you more about that in a later post I think. But quite often in 2D games, you want also your sprites to fade in and out and that’s a problem: creating specific transparency states in your sprite sheets is very limiting or memory consumming. If you use a material property to handle alpha on individual sprites you effectively create duplicate materials which is not good. But hey, we already have the solution: vertex alpha! In the previously described surface shader, we saw that our alpha was multiplied by the alpha component of the vertex color. To make some of the objects fade in or out without creating new materials, we just have to animate the vertex alpha value of each and every vertex of these objects. For a sprite in a 2D game, that’s only 4 vertices per update and per sprite.

Now let’s study a final example, something a lot more trickier used in Transcripted. With this, I’ll try to show you that vertex color is also a usefull way of editing and storing things which have pretty much nothing to do with colors. We will use vertex color for… a complex animation effect!

Here is an animated heart-like organ which is part of the chapter 4 boss in Transcripted. Like pretty much everything else in the game, it’s was modelled in 3D and mapped on a plane with a normal map to allow dynamic lighting. For the ‘muscle’ animations of the first boss in Transcripted, I had used prerendered animation frames. It was very expensive in term of memory usage because it required a specific atlas. Since I wanted the heart to be big and I also wished for a fluid animation, storing every step of animation in a sprite sheet would have been just foolish. The only other option was to deform the mesh: an animated deformation simulating the pulsation of a beating heart. I sure could have skinned my plane with plenty of animated floating bones but it would have been a lot less elegant and a lot more time consumming than what I’m about to present ^^

In a beating heart, you’ve got for 4 cavities. Each cavity is bulging around its center at the same frequency but not exactly at the same time. You can roughly describe the movement of each point of the heart by a sine wave along an axis going from this point to the center of the nearest cavity. Still with me ? It means that each point we will animate will have to ‘know’ its own animation direction and its own animation timing. To describe a direction in 2D we need two values. To describe a timing or a delay we need one value. Since we’re animating vertices here we need to store 3 values in each of them. Vertex color is allowing us to store 4 values ranging from 0 to 255 and that’s more than we need here!

You can see below how the heart plane is tesselated (left) and how the vertex color has been applied by hand (right). The final result is a bit hard to decipher for a normal human brain but you’ll see it’s not that complex.

heart_diffuse_vcolor

In the Red channel of the vertex color, we will store the delay of animation or to be more precise the phase shifting along the sine wave of each vertex. In the Green channel we will store the vertical component of the animation direction. Every value between 0 and 0.5 points upwards. 0.5 will be the equivalent of 0 and every value between 0.5 and 1 points downwards. In the Blue channel we will store the horizontal component of the animation direction. Less than 0.5 is pointing to the left, more than 0.5 to the right.

heart_vcolor_channels

With the Vertex paint modifier of 3DSMax it’s quite easy to paint each color channel on a different layer and use an additive blending between those to create the correct final output. Of course your mesh must be sufficiently tesselated to allow a smooth deformation.

Finally let’s take a look at the shader making the heart pulse. Unity won’t understand by itself what you were intending to do with these weird vertex color values! The whole magic takes place in the vertex shader part of the shader :

Shader "Transcripted/heart" {
   Properties {
       _Color ("Main Color", Color) = (0.5, 0.5, 0.5, 1)
       _SpecColor ("Specular Color", Color) = (0.5, 0.5, 0.5, 1)
       _Shininess ("Shininess", Range (0.01, 1)) = 0.078125
       _MainTex ("Base (RGB) Gloss (A)", 2D) = "white" {}
       _BumpMap ("Normalmap", 2D) = "bump" {}
       _myTime ("Time", Float) = 0.5 
   }

   SubShader {
   Tags {"Queue"="Transparent" 
        "IgnoreProjector"="True" 
        "RenderType"="Transparent"
   }
   LOD 400
   CGPROGRAM
   #pragma surface surf BlinnPhong alpha vertex:vert
   // vertex:vert => there's a vertex shader and it's named vert
   #include "UnityCG.cginc"

   sampler2D _MainTex;
   sampler2D _BumpMap;
   sampler2D _LightMap;
   float4 _Color;
   float _Shininess;
   struct Input {
       float2 uv_MainTex;
       float2 uv_BumpMap;
       float4 color : COLOR;
   };

   // our vertex shader, that's where the magic happens!
   void vert (inout appdata_full v) 
   {
      // v.color is storing the vertex color of each vertex
      // we define a phase, a timing for the animation which is
      // the time since the start of the application (_Time[1])
      // plus a delay proportional to the red channel of the vcolor
      float phase = cos( _Time[1]*5f + v.color.r*1) ;

      // we offset the x position of the vertex currently processed
      // by a value proportional to the blue channel of the vcolor
      v.vertex.x+=100*(v.color.b-0.5f)*phase*phase*phase*phase;

      // we offset the y position of the vertex currently processed
      // by a value proportional to the green channel of the vcolor
      v.vertex.y+=100*(v.color.g-0.5f)*phase*phase*phase*phase;
   }

   // a surface shader (very similar to the previous example)
   void surf (Input IN, inout SurfaceOutput o) {
       half4 tex = tex2D(_MainTex, IN.uv_MainTex);
       half4 c = tex * _Color * 2;
       o.Albedo = c.rgb ;
       o.Gloss = tex.a * c.b;
       o.Alpha = c.a ;
       o.Specular = _Shininess;
       o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
   }
   ENDCG
   }
}

That’s it. You can see on the final result that each vertex is moving in the right direction and with the right timing to create the illusion of a beating heart.

It’s not unusual to store custom data in the per-vertex properties of a mesh for something completely different than their original purpose. If you don’t need them for the actual rendering or if you know they have a constant value in your application, you can use vcolor but also normals, tangents and uvs for a lot of things! One of the advantages of vcolor is that you can easily visualize the data and apply by hand custom values using softwares like 3DSMax.

That’s it! I hope I’ve convinced you to take a closer look at what you can actually do with my friend vertex color.

 

6 thoughts on “A game of Tricks II – Vertex Color

  1. Thanks for the post. It was an awesome read for someone like me who didn’t understand much about vertex color. We’re going to try and apply this technique in the game we’re currently developing. Keep up the awesome work you’ve been doing with those posts!

  2. nice! thanks for putting this up. I’m currently learning to write some basic shaders for myself and this will be of great use for me in terms of a learning source I think! best, marcel

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>