In a previous post, I wrote about integrated tests and how they actually decrease your chance to catch regressions, decrease testability of the system and can lead to a test suite where nobody really trusts the results. That post captures some of my thoughts after hearing a talk and reading an article by J.B. Rainsberger about the topic.
So, what is the solution?
Mock Objects vs. Integrated Tests
J.B. Rainsberger presented a simple solution (and this is also the solution that I prefer, most of the time, so I am a bit biased ;) ):
Say you want to develop a class (or module or function) A that uses B and C.
- First develop A without having B or C at all in a test-driven way. As you write tests for A, you define the interfaces of B and C, but as of now, you just stub them, preferably with a mocking framework (because you can then use the mock objects in your tests to verify that A behaves correctly).
- Then, develop B and C in a test-driven way. Look at your mock objects from the first step to see how A uses B and C and create unit tests with that knowledge. Then implement B and C. Create more interfaces and mock objects for the collaborators of B and C if necessary.
- You now have tests for A that show you A uses B and C in a way that you deem correct. And you have tests for B and C that show you that B and C behave correctly if used in a way like A uses them. Now you wire the classes up, and they will "just work".
This process is often called “Outside-In TDD” or “Controller-First TDD” or “London School TDD” or “Mockist TDD Style”.
The caveat here is that this only “just works” when the mock objects used in the A tests (which document how A uses B and C) and the tests for B and C are always in sync. So, when you change a mock object that A uses, you also have to change a test for either B or C. Unfortunately, I know of no automated tool that could help you with that.
Why Not Work Like That?
After J.B.’s talk, I heard someone in the audience say:
That's nice in theory, but this will never work on a real project. We had mock objects. It was a mess.
And I hear that sentiment quite often. There is a huge resentment against mock objects in a large part of the programming community, while the other part loves them. Why is that?
“Our project is a special snowflake”. I have seen this attitude at some of my past clients, and in many different programmers. “We cannot use Mock objects because…”. “We cannot do TDD here because…”. “Agile will not work in this company because…”. “This solution will never work on a real/big/complex/hardware/… project, because…”
The project or company is often not that special. In most of those situation, I got the impression that they tried the thing, failed (because they didn’t really try it, made some basic mistakes, didn’t change their attitude/behavior or didn’t seek out for outside help), and now blame their their failure on the thing they tried.
This is what I call “The Mock Objects Trap” when talking about TDD with mock objects.
The Mock Objects Trap
The mock objects trap happens when you use mock objects in a certain way. (If you are not sure what mock objects are or want to refresh your knowlege about test doubles, stubs, fakes, mocks, etc. , read Mocks Aren’t Stubs by Martin Fowler before reading on…)
Mocks to Make the Untestable Testable
Mocks can help you to make untestable code testable.
Say you have a class that cannot be instantiated or tested on it’s own because it has so many collaborators. And you cannot test the correct behavior of the class because there is no accessible state - The class just calls some methods of it’s collaborators.
With a mocking framework, you can still test this class: You stub out all the collaborators, and add expectations or verification for the expected interactions (Note that this is a different use of mocking frameworks than I described in “Outside-In TDD” above). You can now test this class. But, as we will see later, if you are not careful, this can be the first step into the mock objects trap.
Mocks to Cover Up for Bad Practice
Recently I have read an article called “All Your Mocks Are Evil!!”. It’s a long rant about how mocks are bad, because people use them to cover up all kinds of bad practice.
Say you have some business logic in your data access layer. Without mock objects, this would be hard to test. But with mock objects, you just stub / verify some database calls, and now you can test the business logic. The problem is: This business logic does not belong here! But the tests are there and they are green, so this will probably go unnoticed.
Mock objects can be used to hide architecture and design problems. This is step two into the mock objects trap.
Mocks that Lock Down an Implementation
Some tests use mocks in a way that the test creates a lock on how the code is implemented.
Now you cannot change the code anymore without breaking the tests. Even if you did not change the functionality! This means that every time you refactor some code, you will break some tests.
Your tests should be a safety net for refactoring, where the tests stay green when you got the refactoring right (i.e. when you changed some code without changing the behavior of the system). But now your tests have become an impediment to refactoring! This is the final step into the mock objects trap.
Why is it a Trap?
This is a trap because it happens to many programmers when they first try out mock objects. Because they are just starting out with mock objects, they don’t notice the problems until it is too late. And then they hate mock objects and never touch them again. Some write angry blog posts. The story goes like this:
The programmer is working on a legacy system and they want to write automated tests for a specific aspect of that system. The problem is: All involved classes are really hard to test. They have dependencies everywhere and you cannot instantiate them on their own.
So the programmer downloads Mockito and stubs out all the dependencies. She also downloads PowerMock because she wants to stub some static or final methods.
Because the system under test is still very hard to understand and to test, the programmer writes coarse-grained, unfocused tests. And because it is not clear which details are important (it’s legacy code, after all!), she verifies all the interactions the code has with the mocks.
Now we have a system that actually got worse by adding tests with mocks: The tests are unfocused (and thus hard to read and debug). And they lock down the bad practice that exists within the system: When you try to refactor anything, multiple tests will break, even though you didn’t change any functionality.
And this is the first and only experience many programmers have with mocks. This is the “Mock Objects Trap”.
To Recap...
Tests should put a positive pressure on our design. They should help us refactor our code by providing a safety net. We can achieve this with or without mock objects.
But when we are not careful, the mock objects we use might actually hide some design problems or lock down the actual implementation of a class, not it’s visible, desired behavior. Then we have created an evil unit test (N.B.: You can create accidentally create evil unit tests without mock objects too).
The “mock objects trap” is a trap because many programmers unintentionally create such evil unit tests when they first use mock objects. But they only find out about the problems with those tests later - When they have many of them and when the tests have become an impediment to refactoring.
And many of those programmers then blame mock objects and won’t use them again - not even in situations where they would make sense and provide huge benefits.
Have you experienced the problems with mock objects I have described above? How can I help you to overcome them? Please tell me!
