Alternate title: Making Flexes – An After Action Report
This weekend, I released Dr. Schwaiger, a scientist from Sniper Elite V2. The selling point of the release is that he has a full set of highly emotive facial flexing that uses the familiar L4D style controllers. I said I’d be discussing a ‘what I learned’ from it in hopes that if I document my mistakes, someone else won’t make them as well.
Project goals were simple – take a head from something, create flexes, get it working in source. Optional objectives were to learn and utilize DMX (valve’s new file format) to generate stretch/compression maps aka wrinklemaps, and to have the face at least utilize the valve flex systems in place to allow for auto lipsync in SFM. If you watched the 5 second clip, then you know I met all basic objectives and at least one of the secondaries, not a bad day in my book.
Generating the Flexes from a Template
Making flexes is tedious work. Anyone that can grab at dots using a soft selection tool and move them around can do them, but it’s rarely done, and even more rarely done using vlave templates. I had the intention to create the flexes in a way that uses valve’s complex, mathematically generated controller sliders. Why? There a few solid reasons.
SFM ‘gets it’ – I know I preach Source Filmmaker like it was the second coming, but when things work in that program, they do tend to work great. SFM automatically groups flexes it recognizes into handy sub-categories, and that means filtering things you do and don’t want to look at quickly and easily. More importantly, linking up with phoneneme extraction tools gives a great ‘first pass’ lipsync. It’s raw, but it’s great to start with. You’ll also get sliders for sounds like ‘aa’ or ‘ph’ or ‘s’, which is way more intuitive then controlling the base controllers for lip movement.
It’s quality work – The valve flex controller system was professionally made. It catches all kinds of bad combos that could distort a face past the realm of real movement (aka ‘breaking’ them). For example, using valve math, you’ll never be able to do a ‘cheek puff’ and an ‘open mouth’ flex at the same time – which makes sense – you need to have your lips closed to puff your cheeks. They are incredibly subtle fixes and base shape combinations that clearly took a great deal of time to perfect, why throw out that work if you can leverage it?
There are starting resources – Between the VDC and the content available through the L4D/TF2/SFM SDK kits, you have a great deal of template heads and qcs to reference.
But if it’s so great, why doesn’t everyone do it?
It’s complex – the base shapes (aka what you actually need to make by hand) are not exactly easy to create, many require complex vertex manipulation, and there is a sizable library you need to make in order to get it working properly.
It’s all or none – You have to do all of them or it just won’t work well. The system uses everything available to it to generate a single controller, most use dependency chains that can pull data from up to 10 base shapes.
It’s hard to debug and tweak – Don’t like how valve did a flex or don’t like how the end result looks? Good luck shaving away the code to figure out what vertices on what base shape you need to change. That said, if you just stick with the sample heads, you’ll get a nice result the first time.
For this project, I used L4D Bill’s (and occasionally Francis’) facial movements as a base – I manually converted the .qci’s flex controllers to directly control the base shapes so I could look at the flexes in modelviewer (and I even posted a download link to those files in my last post here). Alternatively, Francis and Zoey are both available as a .dmx files in the SFM files – look for them in the lua subdirectory of the SFM sdk tools. If you have a valve approved version of maya or blender installed, you can back convert these to FBX and open those up in max. All of the flexes are stored as morph targets on the base head and you can use those as a basis a flex by flex reference for the base shapes.
As for actually making the flexes, I’d simply copy and rename the base, emotionless head mesh and manipulate vertices on the copy to match the base shape I was recreating. VDC has a nice step by step article for max, I’d suggest you take a look if you’re going to take the plunge. Once the 40 base shapes are made, you’ll need to apply them the default head as a morpher modifier. IMPORTANT NOTE: If you need to change the number of vertices of either the parent or modified mesh (ie you find that the lips weren’t properly welded and you need to go back and weld them) – DON’T DO IT. Morphers only work for exact duplicates of the base mesh. You need to make sure the last thing you do is make the flexes. Confirm that the head is otherwise ready for the game and free or errors – you could very well have an ‘oh shit’ moment when you welded one vertex pair on the base mesh and now the 40 child meshes you made are essentially worthless. There are ways around this though, and that brings up my next topic.
VTA vs DMX
So with the inclusion of the new command line tool in SFM – fbx2dmx – max users have the ability to make heads using the dmx format. This means that max users (me) can implement high end features like wrinklemaps and corrective shapes. There are a few caveats of using this method right now though.
Exporting to FBX can cause problems with the way you export materials – I’m a fan of the ‘use material names instead of diffuse textures’ option on my smd exporter, it gives me greater control over my material library and naming conventions. You don’t even get an option similar on an FBX export, it’s the main reason I’m not writing about my experiences using DMX for a head for the first time.
DMX is less universal – SFM’s fbx2dmx cannot be compiled using SDK2013 studiomdl. This is the default I use because I know the model work for my two target markets (gmod and SFM). I’m going to dig around and test this with more studiomdl and fbx2dmx tools before I chalk this up to a defeat though.
DMX files are not plain text and cannot be easily edited – that’s not to say there isn’t a way to change them after the fact, but in my opinion, dmxedit is a rough around the edges command line tool that is a pain in the ass to use, and you really need to be ready to learn lua or python to make full use of it.
You loose clarity of control as everything is all packed away into a single, hard to access file – Exporting a head with flexes via dmx means you no longer have a vta and your initial definitions of existing flexes clearly in the qc, this means you need to do all your flex name matching in the morpher modifier if you’re going to go with valve sliders. If you want to see what a post DMX qc looks like ,I’d suggest getting yourself a copy of the L4D2 SDK and looking at those resource files.
You can’t cheat as much with face flexes – One of the beauties of separating the flex definitions from the model is that you can tweak the model after you’ve made the flexes. Say you did need to weld some vertices, chances are your .vta will still work with the modified mesh. This is the definition of a double edged sword though, you can make the changes, but you might break the flexes. If you move a vertex outside of the target range, it might not ‘catch’ – that is the vertex that is supposed to move doesn’t, or a vertex that isn’t supposed to move does. This commonly happens around the mouth, you might get bad stretching around the lips or they might stay completely sealed.
One other caveat of this process is slapping more meshes on top of your base, using the ‘defaultflex’ control can damage the portions of the smd that weren’t in the flexfile or were changed after the vta was exported. In the image below, the geomentry for the eyes were not part of the model I did the editing on and the default flex broke them. I also talked about this way back in my scifi citizens article if you want more examples.
Since DMX relies on the morphers as the root of the flex animations and their name in the qc, you don’t need to even touch the timeline/keyframe tools in max. This is a huge improvement over vta files. Since your base mesh can move freely in max without breaking morphers, you can easily and painlessly adjust where the head is in xyz coordinate space. The same absolutely cannot be said about vta files. Once you bake animations and export the file, the head is locked in that spot unless you’re up for editing a lot of code. I wrote a tool that edits vta files for simple translation, but that’s really a hacky way of doing it.
With a bit of education on the subject, you can unlock the added features in DMX that you’ll never be able to do with smd files. For texture artists, that is wrinkle and compression maps, for modelers, that is blend shapes. VDC once again has articles on both, but I’ll boil it down to layman’s terms.
Wrinklemaps are extra textures that have additional face scrunching and stretching in the texture, in the same way normalmaps add extra detail to a model, these add extra detail to a face. If you raise your eyebrows, you can feel the wrinkles on your forehead – these are fine details and movements that just aren’t feasible to model on a game targeted mesh. Using LUA and finely tuned code, however, dmxedit can simulate when these wrinkles would exist based on how much the vertices move. So if you were to hold your finger near your eyebrow and move your eyebrow up, your finger just moved the better part of an inch with the skin. Essentially the wrinklemaps say how much your finger moved at that point and blend the base texture with the wrinkle texture off of that movement.
Corrective(blend) Shapes are another high-concept feature that can be broken down and explained. In many ways, it’s the next logical step from valve’s math derived ‘break’ system, avoiding situations where one flex + another flex really just looks bad. Blend shapes work pretty much automatically based off of your naming convention, and use the underscore as a combinative pair control character. So if AU1 controls eyebrows going in one direction and AU2 controls them going in another, and when you use both at the same it overstretches the center, you can make another flex called AU1_AU2 that fixes this and will kick in when both are selected. This is not limited so single pairs either, they can kick in for more complex combos as well. As described in VDC, these are contextual, and hundreds can be made for a wide combination of flexes on a ‘if you see it break, fix it’ process. Interestingly, the last few controllers in the L4D base shapes are essentially proto-corrective shapes that are combinations of some of the base shapes for eyebrows. These override the single flexes if both are effectively on, but the implementation is via qc math. It stands to reason that the valve artists wanted this feature to be more robust and expanded and automated the functions with dmx.
Honestly though, these are both professional level features, and *most* end users simply won’t care if one flex breaks another in certain situations or the fine wrinkles on the forehead are visible or not. That’s not to say they’re useless additions; they both provide even further control for authors to improve micro details and repair micro issues. In my opinion however , they, and dmx availbility to the community as a whole, have been really overhyped. You can still get great results with smd and vta combinations, and the documentation for the older process is more complete and easy to follow. DMX is the next step, but a good bit of it is still ‘hidden’ to casual modders like myself.
At the end of the day, with this project, I chose the old route. I may know a good bit about how the systems works, but until now, I’ve never actually carried through and did it myself. I learned a ton; you need to fix the mesh before you start making flexes, fbx export can break my material system, and with the right amount of time, you can get a quality result by using Valve’s base code. Next time, I am going to make a set of flexes with DMX and really get a better understanding of that system. If doing something start to finish proves to be better overall, I will be happy to admit I was wrong.
So Dr. Schwaiger is out in the wild. You can get him here, and he is both compatible for gmod and SFM (which, if I chose the DMX route, may have not been the case!)
Those maps from Black Snow? Still working on them. I did a good bit of work on the interior this weekend, mostly deleting sprites that didn’t transition well, changing model properties to allow for rendering in SFM, removing unneeded game logic, and kicking up lightmaps. I do ahve a rough blocking of my interior preview video and with the less then stellar SFM exports I’ve been getting, I’m going to try and turn to an external tool to do some editing. I’m using photoshop right now, but it’s like bolting a hitch to a Ferrari to haul a trailer.
Here’s a preview video: