Unit Testing

ID:121373
 
Nov 27 2011, 11:10 am (Edited on Nov 27 2011, 11:20 am)
You've written some software or a game. You go to add a cool new feature that interacts with previous features A, B, and G. Little did you know, you broke feature B to some extent in the process. Three months later, your client points out to you that feature B isn't working properly. You move, frantically, in an attempt to fix feature B to work properly again before your client becomes overly dissatisfied.

This scenario could have been prevented if you had a good system for regularly testing incremental changes in place. If you haven't read Deadron's article on unit testing in a while, I highly recommend you do so again. It's a good read and it can help remind you why test-driven development is a great way to do things.

For BYOND, Deadron.Test is a great test framework. It will help you feel encouraged to switch to test-driven development with its ease of use and lack of required intervention. With Deadron.Test, it's automated for you such that you just have to make an /obj/test/verb and your test is already ran.

For C++, I wrote a similar testing framework inspired by Deadron.Test. The process for using it is like so:

  • Create a new set of header and source files for your test(s).
  • Create a new class derived from TestFramework::UnitTest
  • Define the public `bool Run(void)` function. A return of 'true' indicates success, a return of 'false' indicates failure. You can use Message(const char *) to supply a message to go along with the failure message.
  • Define `const char * Name(void) const` to return a name for your unit test.
  • Use the `DefTest(unitTestClass)` macro on your class in the source file.
  • To make the unit tests actually run, create a new `TestFramework::Core` instance and call `bool Run()` on it. A return value of true indicates success on all tests, a return value of false indicates a failure. It outputs which tests pass and the test that fails (if one fails).
  • Make sure that testing.cpp is compiled last
  • Optional: If you wish to create a new variable of some type belonging to your testing class, you can define `void Clean()` to do any necessary clean-up after your test concludes. This can be an easy way to avoid the redundancy of cleaning up in the several scenarios that the test may fail without causing any leaking of any sort.


Other recommendations:
  • Do not put stack-based variables outside of Run(). The way the testing framework is setup, your test class gets constructed inside of a static global (allocated at program startup), which never gets destroyed.
  • Create two builds for your application, one to function and one to run the unit tests. I personally use CMake and have it build the normal application without the testing framework then a second version with "TESTVER" defined and my testing framework included. My `main()` is one giant #ifdef/#else/#endif. You can do it however you want, but I do not recommend including the testing framework in a release build.


After months of usage and refining, I have simplified the process of using it as much as possible. After you've got it running tests and you've written a test class or two, it starts to come naturally to you.

Using a test-driven development cycle has saved me from going bald, and it could save your hair too. There have been countless times where I've written up a new piece of code to interact with something else only to find that that something else didn't completely operate the way I wanted it to because of some changes I made else-where. Now that I know how to write unit tests, this happens no more.
Nov 27 2011, 11:16 am
Oh. Thanks for this post! I'll reread Dead's article on Unit Testing.
Nov 27 2011, 10:40 pm (Edited on Nov 27 2011, 10:47 pm)
It's actually not redundant at all, even for small projects. For an example, let's look at Breakout. Breakout is a reasonably simple game- you have a ball, a paddle, blocks, and some basic physics.

What's worth testing, you ask?
Collisions, making sure blocks are destroyed properly and the ball rebounds properly after colliding with wall/block/paddle.
Power-ups, making sure they actually modify everything properly.
Score, making sure score is being added properly when bricks are destroyed.

But why is it worth testing? Good question. For the scenario where you have one level, you can play the game out within a few minutes and it's alright, so it's not really that big of deal.

But what about if you add a level, two levels, or even n levels? What if for any given level m, you introduce a new block that can absorb exactly m collisions before it dies? Then you have to spend even more time going through and making sure that everything works properly. Hitting a block m times isn't too bad, but keeping track of which blocks you've hit how many times can be irritating.

The main point behind that is that as your game grows in complexity or length (in playing time), you have to spend even more time testing it. Now, that additional time it takes to test the new features may be proportional to the square root of the time it took to implement the new feature, or it may be directly proportional to the time it took to implement the new feature, but what if you add something else and have to change it? Then you spend that time all over again going back to test it, and it's even more painful if certain conditions have to exist. Instead, you could have written a test that emulates all of these conditions and makes sure that the feature works properly, and you can know as soon as you start up the game that it works exactly as you intended.

Does it increase your workload? Yes, it does. You have to go through and think of different ways to use your function that might possibly break it now and in the future. You just have to ask yourself the question, "Would I like to take a little time now and write a test for this that runs each time I start the game to save me a lot of time, or would I like to spend a lot of time testing that feature?"

--

Looking back at my comment, I realize that it's a convoluted mess. I feel that writing unit tests is especially important for even a BYOND project because of the selection of people that develop for BYOND. The main age range of developers on BYOND right now is probably ~14-19. Most of them are probably in High School, so they don't actually have a lot of time on their hands. Their development on BYOND is just a hobby that they do in their spare time. Writing unit tests for every feature will increase their workload, but it will also save them their valuable time now and in the feature by not having to go through and re-test every feature individually. I compared `BYOND projects` to Breakout because Breakout is a perfect example of a game with minimal features, usually less than the average BYOND game.
Nov 28 2011, 2:17 am
Easily, actually. The collision between the ball and the wall doesn't actually require any rendering what-so-ever. They're both just as simple as datums or classes or what-have you, with velocity and angle as variables. It wouldn't be too far-fetched to setup a ball and a wall and simulate a collision between the two. All that you really care to check for is the instantaneous velocity directly after the collision as a result of the acceleration after hitting the wall.

In Deadron's article, he explains how he wrote Last Robot Standing with unit tests to ensure that all of the cards interacted as they were designed. Complex interactions are exactly what unit tests work amazingly for, as you can simulate them better in code faster than you could by playing it out.
Nov 28 2011, 4:25 am (Edited on Nov 28 2011, 6:29 am)
Audeuro wrote:
Complex interactions are exactly what unit tests work amazingly for, as you can simulate them better in code faster than you could by playing it out.

Unit testing can't test complex interactions because those interactions are at a level larger than the unit. Unit testing is when you test an individual unit, but those complex interactions only occur when units are combined. It's possible to write code to help test a game, but what you're talking about isn't really unit testing, it sounds more like integration testing.

These terms are "one size fits all" processes that are never perfect but often decent. You have a brain, decide for yourself what testing methods are appropriate for your project. Games are often difficult to test because it's difficult to write code to determine if a test passes - sometimes you just need to run something and see how it works. Suppose you have damage numbers pop up when an enemy takes damage. When you test this you want to make sure it works (that the numbers do appear) but you also want to make sure the numbers stay for an appropriate amount of time. You can't write a test case that checks if the duration is appropriate*. You just have to set your game up so you can easily access all parts of it.

* edit: you'd also have trouble testing if the numbers appear properly because this can depend on many things - icon, icon_state, layer, etc. Actually determining that the numbers could be seen would be difficult to test with code.

For example, suppose you're making a typical RPG. When a player starts the game they have to create a character and watch the intro cutscene before they get to the actual game. If you want to test combat and leveling up, you don't want to have to go through that each time you start the game. Make a special Login() proc for testing that gives you a character already on the map who is 1 experience point away from leveling up and will get a fight on their next step (you could even give them an excellent weapon so they'll kill their opponent in one shot).

It's not about using a "test-driven development cycle" or any other buzzword, it's about using your brain and your programming ability to make game development and testing easier for yourself.
Nov 28 2011, 11:29 am
Forum_account wrote:
Unit testing can't test complex interactions because those interactions are at a level larger than the unit. Unit testing is when you test an individual unit, but those complex interactions only occur when units are combined. It's possible to write code to help test a game, but what you're talking about isn't really unit testing, it sounds more like integration testing.

Fair enough- I'm guilty of escalating to another scope in my tests.

[...] Games are often difficult to test because it's difficult to write code to determine if a test passes - sometimes you just need to run something and see how it works.

True, but you can still write tests for a lot (the majority, usually) of your game to save you quite a bit of time in the end.

Suppose you have damage numbers pop up when an enemy takes damage. When you test this you want to make sure it works (that the numbers do appear) but you also want to make sure the numbers stay for an appropriate amount of time. You can't write a test case that checks if the duration is appropriate*. You just have to set your game up so you can easily access all parts of it.

* edit: you'd also have trouble testing if the numbers appear properly because this can depend on many things - icon, icon_state, layer, etc. Actually determining that the numbers could be seen would be difficult to test with code.

The first one is true, and I tried to hit that point with a previous comment- you can't actually systematically test the rendering of features. You could write a test to ensure that the layering is right, but that's getting unnecessarily anal about it. As for the second part, about the duration. I'd argue that you could actually test this, although the design of your test in DM could/would be shady- the proc that deals with taking damage would have to draw the text on the screen and retain a handle to the text (however it was drawn) on the screen. The test could then retrieve this (a separate proc, obviously, not the return value from the damage proc) and check that it's deleted after exactly n minutes.

For example, suppose you're making a typical RPG. When a player starts the game they have to create a character and watch the intro cutscene before they get to the actual game. If you want to test combat and leveling up, you don't want to have to go through that each time you start the game. Make a special Login() proc for testing that gives you a character already on the map who is 1 experience point away from leveling up and will get a fight on their next step (you could even give them an excellent weapon so they'll kill their opponent in one shot).

It's not about using a "test-driven development cycle" or any other buzzword, it's about using your brain and your programming ability to make game development and testing easier for yourself.

I don't think you can really call it a "buzzword" yet (Google only returns ~2.89e6 results, as opposed to the ~10.3e6 for "object oriented programming"), but regardless: BYOND developers could benefit from looking at their projects from the perspective of "Alright, this feature should be testable. How can I write it so that it is testable?" assuming they can actually get anything done, it should at the very least improve the design choices (to some extent) of the developers on BYOND.
Nov 28 2011, 12:14 pm
The test could then retrieve this (a separate proc, obviously, not the return value from the damage proc) and check that it's deleted after exactly n minutes.

You can automatically verify that the object existed for as long as you told it to exist, but you can't automatically verify that the amount of time you told it to exist is appropriate (long enough that you can read the numbers, but not too long that it looks bad).

If you're going to have to manually test a feature to make sure these other aspects (the ones that can't be automatically tested) are ok, does it save you time to have the automated tests? It's rarely mentioned when people talk about testing, but you do have to make sure your tests are correct. The time to create, test, and troubleshoot a test case to verify how long the damage numbers exist for could be significant.

In unit and integration testing there's an emphasis to test *everything*. That's fine for a huge project where you have a team of people devoted to testing, but that part of the concept doesn't carry over to hobby game development.

Alright, this feature should be testable. How can I write it so that it is testable?

I guess I partially agree. Good code ends up being reasonably testable, I'm not sure I'd encourage people to make a conscious effort to think "how would I test this?" as they program. I do think it's important for people to realize that they should make an effort to test and use their programming ability to simplify and improve testing.
Nov 28 2011, 2:29 pm
Forum_account wrote:
You can automatically verify that the object existed for as long as you told it to exist, but you can't automatically verify that the amount of time you told it to exist is appropriate (long enough that you can read the numbers, but not too long that it looks bad).

Aesthetics is something that will always require human interaction.

If you're going to have to manually test a feature to make sure these other aspects (the ones that can't be automatically tested) are ok, does it save you time to have the automated tests?

Most of the time, yes, it would save you time. For each specific unit, the amount of time saved would be negligible, but over the course of an entire project? It adds up. As you mention a few paragraphs later, it's mostly useful for large teams with personnel dedicated to testing, but another thing to consider is the type of people that make up BYOND developers. As I mentioned earlier, most of BYOND's developers are probably in the 14-19 age range, probably programming as a hobby. Out of all of those developers, how many might have a dream of becoming a big-time game developer? It might be a useful mindset that aids in their professional development.


It's rarely mentioned when people talk about testing, but you do have to make sure your tests are correct. The time to create, test, and troubleshoot a test case to verify how long the damage numbers exist for could be significant.

I like to provide test cases for the test cases to make sure they're working properly. :)

I guess I partially agree. Good code ends up being reasonably testable, I'm not sure I'd encourage people to make a conscious effort to think "how would I test this?" as they program. I do think it's important for people to realize that they should make an effort to test and use their programming ability to simplify and improve testing.

For most people on BYOND, it's at least a step away from the absolute wrong direction. Is it the best path to take? Not necessarily, but taking on a mindset of "How would I test this?" while they're programming isn't generally going to be as detrimental to their professional development as the path that most of them are on now.

As a side question, is there any specific reason that you chose only these two particular parts of my previous comment to comment on? Mainly, if everything else just wasn't worth commenting on then I'd like to work to correct that.
Nov 29 2011, 4:19 am
Audeuro wrote:
For most people on BYOND, it's at least a step away from the absolute wrong direction. Is it the best path to take? Not necessarily, but taking on a mindset of "How would I test this?" while they're programming isn't generally going to be as detrimental to their professional development as the path that most of them are on now.

I think there needs to be another step first. To write test cases you have to look at code and think "what could be wrong with this?" and I don't see many BYOND users who are willing to think like that. It's a good skill whether you apply it by writing unit tests or not, so I'd encourage writing test cases if it gets people to think that way (even though I don't think the tests themselves are that useful). I'm just not sure that BYOND users would be very receptive to that.

I think people would be more receptive to non-automated forms of testing - how to write code that lets you be able to easily put a player into any game situation you need to test without interfering with the rest of the game. It's a lot easier to see the benefit and it's the kind of testing you'd need to do anyway. Instead of saying "go out of your way to write these test cases, I promise it'll pay off" you'd be saying "here's how to implement the same thing in a different way that makes it easier to test".

As a side question, is there any specific reason that you chose only these two particular parts of my previous comment to comment on? Mainly, if everything else just wasn't worth commenting on then I'd like to work to correct that.

It was worth commenting on, I just didn't have a comment for it.
Nov 29 2011, 4:33 am
Forum_account wrote:0
I think there needs to be another step first. To write test cases you have to look at code and think "what could be wrong with this?" and I don't see many BYOND users who are willing to think like that. It's a good skill whether you apply it by writing unit tests or not, so I'd encourage writing test cases if it gets people to think that way (even though I don't think the tests themselves are that useful). I'm just not sure that BYOND users would be very receptive to that.

I can agree to that.

I think people would be more receptive to non-automated forms of testing - how to write code that lets you be able to easily put a player into any game situation you need to test without interfering with the rest of the game. It's a lot easier to see the benefit and it's the kind of testing you'd need to do anyway. Instead of saying "go out of your way to write these test cases, I promise it'll pay off" you'd be saying "here's how to implement the same thing in a different way that makes it easier to test".

And this is a scenario where use of the preprocessor for more than just macro-constants would come in handy, but it seems as if very few want to dive into the conditional compiling, despite how simple it actually is. Out of all of the projects on BYOND that I've looked at, very very very few take advantage of conditional compiling.
Nov 29 2011, 4:50 am
I'd avoid using the preprocessor because it tends to complicate things unnecessarily. I'd rather put testing-related code in a file that I can include or uninclude to enable/disable testing rather than remember which compiler flags I need to set to enable testing.

There are also lots of simpler things that are just about how to write code that can easily be tested.

I'd bet that lots of people have their login/character creation code started right from client/New() or mob/Login(). If you want to test combat you'd have to log in, load a character, go to an area with enemies, and get in a fight, or, you could modify your game's Login() proc to bypass the loading and place your character in a place with enemies. If you structure the code differently you can easily override parts of the login process to bypass character creation without actually modifying the game's Login() proc - you can test it easily without making any changes to it.
Nov 29 2011, 9:55 am
There's an abundance of ways to do it and it really just boils down to personal preference- I'd prefer using the preprocessor because it allows me to keep the testing code within a range of my normal Login() or New() behavior, serving as a reminder that I need to do testing if I'm other-wise forgetting.