Test Driven Development (TDD) seems to be quite popular, especially in web application development. But, we have had zero test code in Handmade Hero. Is test code not commonly used in game development, or is it just the case in Handmade Hero? Why is this so?

I don't pretend to speak for Casey, but I will give my answer to why I don't think TDD is very useful in game development. TDD practically requires everything to be decoupled and every operation must yield a verifiable result. How do you verify that a polygon drawn to the screen is drawn exactly to specification? To achieve any kind of code coverage for your tests, you would have to create these huge abstraction layers for every facility you use, which costs a lot in performance, readability and overall sanity. The web development crowd has invented huge infrastructure to deal with that which in my opinion leads to a great deal of bloat and complexity.

Casey's way of writing usage code first when designing a part of the APIs is somewhat related to TDD with its Test First principles.

Regression tests are not inherently evil, as they can be used effectively for smaller subsystems and libraries which you make use of, but to apply it across a whole game seems really difficult and intrusive.

It would be interesting to hear other opinions on this subject.

I also don't speak for Casey, however, remember that so far, he's been working entirely on the platform API, and that's the one bit that you can't automatically test well.

It's more practical to automatically test higher-level code, because you can stub or mock the lower-level API. It's not uncommon for retargetable game engines to have a Direct3D backend, an OpenGL backend, and a stub backend.

I think that it's quite alright to have a stub implementation of the platform API. I wonder how many teams actually follow TDD fully which implies always writing tests for everything before implementing it and furthermore letting the tests guide all development of production code. I am not convinced that's all that common. Even some web development gurus that used to buy into the TDD-methodology completely like David Heinemeier Hansson, inventor of Ruby on Rails, are moving away from it, practicing it more selectively. TDD is dead

Here is an entertaining talk by Erik Meijer, co-creator of the Rx-framework for .NET and HACK-language (not suitable for children, plenty of colorful language included) about some of the common methodologies used in the software industry:
One Hacker Way

The configuration space of a web program tends to be drastically simpler than that of a game, so "test driving" the development is more feasible in web scenarios. Like previous respondents have pointed out, web programs are usually easy to decouple into unrelated parts with small configuration spaces that can be reasonably tested as units.

I would assume that is most of where the disparity comes from. Games can usually only do randomized testing, which is good to do, but which can "drive" development, it can only be used as an "extra beta tester".

- Casey

Also useful is the classic 1986 paper by David Parnas (who else?) and Paul Clements, A Rational Design Process: How and Why to Fake It.

http://users.ece.utexas.edu/~perry/education/SE-Intro/fakeit.pdf

There is often a performance and code size hit associated with decoupling things. There are ways to work around that (oh the macro and build system hacks I've seen :( ) but then you are making your design and implementation more complex than it needed to be (see the "TDD is dead" link) which might not be a good trade off.

In the work I currently do (embedded - litte memory, crappy CPU) I have found that a "design by contract" type approach (think lots of asserts) gets me most of the benefit without as many downsides.

For an alternative point of view, I can recommend the book "Test Driven Development for Embedded C". Even if you don't care about embedded it is the best TDD book I've read for C/C++.

rathersleepy

In the work I currently do (embedded - litte memory, crappy CPU) I have found that a "design by contract" type approach (think lots of asserts) gets me most of the benefit without as many downsides.

I do a lot of audio processing soft real-time code for ordinary PCs and workstations with C++ (no allocations except at start-up, no exceptions, lock-free concurrency only) and I also find that "design by contract" is the way to do it. I think multi-level asserts are really useful, you have for example three levels of asserts: very cheap (can be enabled most of the time), medium (can be disabled when full performance is needed but enabled most of the time during development, and expensive that you enable occasionally to really test your code. Then you can select the level of assert-checking with a #define globally or locally for the build. The expensive asserts can be like "is_sorted" and other stuff that requires quite a bit of work to test pre/post conditions of your algorithms.

Then you probably want to combine this with fuzzing for sensitive components such as the msg queue used between threads and other key algorithms/data structures.

CPPCON: Defensive Programming Done Right

Games being more complex is actually an argument for more automated testing, not against it.

And it seems quite feasible given Casey's clever dependency inversion, where the game acts as a service to the platform layer (instead of the other way around). That makes writing many kinds of tests of the game code relatively simple. Set up a game state, call the frame update function, and check the resulting state.