You are here


Boilerplate vs. Magic

Just about a month ago, Phil Webb tweeted:

What would you like to complain about?

[ ] Too much magic
[ ] Too much boilerplate

Phil Webb - @phillip_webb

and just a little bit later, Josh Long tweeted:

“too much magic” = you haven’t read the docs on @SpringBoot ’s auto-configuration

Josh Long - @starbuxman

Here's my take...


Magic means that things happen at runtime that I cannot immediately explain just by reading the code. At least to me. I know, this definition is quite fuzzy. There is always something that happens at runtime that cannot be expected by reading the code, even if you program assembler (processors are weird). The difference between "not magic" and "magic" here is just "How big is the surprise?".

But I would definitely consider a method that violates the "Single Responsibility Principle" or "Command / Query Separation" to be magic (from the caller's point of view):

public User getUser(final long userId) {

    User user = userRepository.get(userId);

    return user;

When you just read code that calls this method, you will never expect that this method changes the user in the database. But you can just open the code of the method and find out. So the magic here is not quite strong (i.e. on a very low level).

Here's another one:

@Transactional(propagation = Propagation.REQUIRES_NEW)
public void persistLastAccessTime(User user) {

Now the method is reasonably named. It sets the access time and persists the user. It still does two things (violating the SRP), but they are somewhat related. And it always runs in its own transaction.

The magic that happens here cannot be found in the code itself anymore. The magical thing here is how Spring implements the Transaction - Especially the "REQUIRES_NEW": It suspends the current database connection and opens a new connection for the new transaction. It's the only way REQUIRES_NEW semantics can be implemented with most (if not all) relational databases.

Your code will work fine in your unit tests, it will work fine in your integration tests, it will even work fine when you test it on your staging server. But then you start a load test, and everything breaks down. What has happened? You didn't carefully configure your database connection pool size to be much bigger than your thread pool size, and you ran out of database connections.

This doesn't have to happen. If you read the documentation really carefully, you can find out that you will need much more parallel DB connections with REQUIRES_NEW than without. But the point is: To find out what's really happening, you have to read the documentation and know what that means for your application, or you have to load test. This is a stronger kind of magic.

One last example, again with Spring, but this time it's Spring Boot:

public class DemoApplication {

	public static void main(String[] args) {, args);

This is the complete source code of a demo application. When running this class, Spring Boot will start an in-process web server. If we go with my definition from above, this is definitely magic.


Boilerplate code is code that you have to write again and again. You'll write it in every application you work on. Sometimes you even have to write it multiple times in the same application. It's the boring code, that nobody really likes to write.

Boilerplate, Magic

Magic and boilerplate are both subjective. Look again at the transaction annotation example from above... Some developers I know would immediately say "Well, of course this requires a new DB connection" while others would be quite surprised that the defect I described can really happen under high load.

And with the Spring Boot demo application, the whole point of writing the application like this is to have Spring Boot do a lot of work for you in the background. So I, personally, hope that people who write code like this know what they are doing (i.e. that they know Spring Boot well enough).

Magic and boilerplate are not necessarily bad. Consider the Spring transaction magic from above: Yes, writing begin and commit is not really hard. And if you would have done it yourself, you definitely would have spotted the too-many-sessions-but: You would have realized that you cannot implement the "REQUIRES_NEW" on the current session. On the other hand, if you always write "begin" at the beginning of a method and "commit" at the end, this is quite redundant.

Or let's look at Spring Boot: Yes, configuring and starting an in-process web server is not hard - it's just a couple of lines of code. But if you have to configure multiple things (a web server, JPA, REST endpoints, ...), it will become quite a bit of boilerplate and Spring Boot can really help you focus on what is important in your application.

Boilerplate is not the opposite of magic! You can have both in the same code base, even in the same class. You can have a single class where there is a lot of boilerplate and where you cannot explain everything that happens just by reading the code.

And you can also have neither: If you find the right abstractions, you can have really expressive, self-documenting code. Code where everything you need to know about what it does can be found within the code itself. And this code will contain no boilerplate, because you found the right abstractions. This is easier with some languages than with others, but I don't think there is any language which make reducing boilerplate impossible (at least for most boilerplate).

Too Much Magic

Boilerplate is mostly a problem when writing code. Magic when reading & debugging. We read & debug more than we write.

David Tanzer - @dtanzer

Libraries and tools that do magic for us help us to develop software faster. They make the boring tasks go away and help us focus on our business code. That's the theory, at least.

On the other hand, they sometimes make reading and debugging our code harder. You want to step into a method, but can't, since the method was created with a byte code manipulation library. Or you read the code of some method you want to call, but don't see all its behavior, since some aspect oriented programming tool adds behavior at run time.

Libraries and frameworks where I constantly have to read documentation or debug code I didn't write are in this category for me: They make it harder for me to understand my own code. Don't get me wrong: I know that I have to learn and understand the tools I am using. But with some tools, I have to come back to the documentation or open the debugger more often and for longer than with others.

Since we spend much more time reading and debugging code than writing new code, I really don't want too much magic like this - i.e. magic that makes reading and debugging code harder - in my code.

Too Much Boilerplate

Too much boilerplate is worse for me.

I would mostly prefer annotation generation magic over writing stuff myself. If I understand the tools it is less code to read.

Janusz Leidgens - @Killerdackel

Boilerplate is mostly a problem when writing code for the first time. You have to write it, and you have to get it right. But after that, you won't touch that code anymore.

Boilerplate is not so much a problem when reading code. It is often easy to understand and easy to recognize. So you can just skip it.

Since we spend much more time reading and debugging code than writing new code, I usually don't mind a little bit of boilerplate code. But Janusz has a point here: If you have a lot of boilerplate code, you have more code to read and more code to maintain.

What Now?

Check boxes, not radio buttons. *takes hat off*

What would you like to complain about?

[ ] Too much magic
[ ] Too much boilerplate

Samir Talwar - @SamirTalwar

Of course, too much boilerplate and too much magic both are problems. As I said above, boilerplate is not the opposite of magic. You can have both, and you can have none of them.

You can write expressive code that gives you a lot of information about what is happening just by using good names and following some general design principles. And when you find the right abstractions for your program, you will still have very little boilerplate.

If you have a statically typed language, you can write your code in a way that your type checker or compiler tells you if you got something wrong. This will make your code even more expressive and safer. For example, when you use Spring, use the Java configuration whenever possible. The compiler will prevent you from making a big number of mistakes you could make with the XML configuration.

And always try to eliminate both: Boilerplate and magic.

Posting Type: 

6 Reasons Why You Should Avoid Singletons

A little more than a decade ago, I used singletons a lot. I had a shortcut in eclipse where I only had to write "singleton" and press "CTRL+Space", and boom: Eclipse would generate all the necessary code to make the currently open class be a singleton. Since then, I learned a lot. And the tools we have to avoid Singletons (i.e. the Singleton Pattern) have matured. Now I try to avoid the singleton pattern like the plague. Here is why...

Yes, I am really writing a blog about why the "Singleton Pattern" is bad. In 2016. But it comes up in discussions time and again, so here is a list of reasons why you should avoid singletons.

Actually, there's nothing wrong with having only one instance of an object. But using the "Singleton Pattern" to achieve this is never the best solution. Towards the end of this post, I will show you why and how you can do it better.

Some of the reasons why using the "Singleton Pattern" is bad are closely related to others. Like, "Inversion of Control Violation" has mostly the same consequences as "Coupling to Concrete Implementation". But I will list them separately anyway because they give you different motivations to avoid the Singleton Pattern.

The Singleton Pattern

The singleton pattern is a design pattern that you can use when you need exactly one instance of an object. An implementation looks like this:

public class InterestingThingsDoer {
    private static InterestingThingsDoer instance = new InterestingThingsDoer();

    //possible some instance state

    private InterestingThingsDoer() {}

    public static InterestingThingsDoer getInstance() {
        return instance;

    public ResultType doSomethingInteresting() {
        //something interesting

Callers can now get the single instance of the Singleton and make it perform some work for them:

public void evenMoreInteresting() {
    InterestingThingsDoer doer = InterestingThingsDoer.getInstance();
    ResultType result = doer.doSomethingInteresting();
    //do something even more interesting...

Great - Now you can implement your abstract factories, services, facades, ... in a way that guarantees that every caller operates on the same, shared instance. What could possibly go wrong?

Coupling to Concrete Implementation

Singletons couple all callers to the concrete implementation of the singleton, because the callee asks for the singleton object. Your classes become harder to test: You cannot test them in isolation, without also running code from the Singleton.

In the example from above, you cannot test the method "evenMoreInteresting" without also running some code from "InterestingThingsDoer". You cannot test this method in isolation.

Why is this a problem?

Suppose the method from above looked like this:

public void evenMoreInteresting() {
    InterestingThingsDoer doer = InterestingThingsDoer.getInstance();
    ResultType result = doer.doSomethingInteresting();
    ResultTypeDao dao = ResultTypeDao.getInstance();;

Now you need a database to test "evenMoreInteresting()". And if "evenMoreInteresting()" is again defined in a singleton (e.g. because it's part of a stateless service, and everyone knows that you only need one instance of a stateless service), you'll need a database to test every method that calls "evenMoreInteresting()". And so on. A large percentage of your tests will be integrated tests, and those are evil.

Inversion of Control Violation

When you use a singleton in one of your classes, you violate "Inversion of Control": Your class does not get the collaborator (the singleton object) injected, it usually asks for the singleton object.

Why is this a problem?

In the code above, evenMoreInteresting() has complete control over the control flow, because it gets a concrete implementation of all required collaborators and decides what to do with them.

To achieve inversion of control, you have to make sure that classes are not responsible for finding / creating their collaborators (among other things). You do this with a technique called "dependency injection" - Preferably "constructor injection".

But my constructor signatures are getting too big!

If you change some existing code to dependency injection, especially constructor injection, you will notice that some of your constructors have lots and lots of parameters. Take that a chance for refactoring: This class is clearly doing too much!

Open / Closed Principle Violation

Singletons themselves violate the "Open / Closed Principle": If you want to extend or change the behavior of the singleton, you have to change the class. In other words, you cannot change the behavior of "InterestingThingsDoer" from above without opening the file and editing the code.

Why is this a problem?

Changing code is much riskier than adding new code. When you change existing code, there is always the possibility that you'll introduce undesired side effects (a.k.a "defects" or "bugs") in totally different areas of the code that depend on the changed code.

When you can extend the functionality of existing code by just adding new code, the probability that you'll introduce side effects in "unrelated" areas of the code goes way down (but might still be greater than zero, depending on your architecture and design).

So, classes / modules / functions that violate the open / closed principle increase the probability that you'll introduce defects later, when you need to change them.

Single Responsibility Principle Violation

Singletons themselves violate the "Single Responsibility Principle": In addition to their main responsibility, they are also responsible for implementing their own lifecycle.

Classes that use a singleton also violate the "Single Responsibility Principle": In addition to their main responsibility, they are also responsible for deciding the concrete class of (at least one of) their collaborators.

Why is this a problem?

Every responsibility of a class is also a reason to change. You'll have to change "InterestingThingsDoer" whenever you need to change what it does and whenever it's lifecycle changes. And you have to change all callers of "InterestingThingsDoer" when its lifecycle changes, since they have multiple responsibilities too.

And, as already stated above, changing code is inherently risky: If you change code because some responsibility of a class has changed, there is some risk that you'll also break code that deals with other responsibilities of the same class.

Also, if your class has multiple responsibilities, you have to test all of them. And you probably cannot test them in isolation. This means you'll have to write more complicated / hard to understand tests.

Dependency Inversion Principle Violation

The "Dependency Inversion Principle states that the high-level policy of your system should not depend on low-level details. Also, abstractions should never depend on details.

Singletons often implement some low-level detail. When you use them from your business logic (the high-level policy of your system), you violate the "Dependency Inversion Principle".

Why is this a problem?

The dependency inversion principle ensures that you can change the low-level details of a system without having to change the high-level "business logic".

Imagine a vending machine: You select a product, pay the price, get the product. When the implementation of this high-level workflow has any dependencies to low-level details, like how the payment works, you might have to change code dealing with this high-level workflow when you add some new payment method (like NFC payment). You don't want this, because changing code is dangerous.

Increased Coupling / Decreased Cohesion

Often, multiple classes depend on the concrete implementation of a singleton. This increases coupling in your system: When you want to change the behavior of the singleton, you probably have to check and change all the callers. You always have common coupling between all classes that use the singleton: They share a global variable - the singleton!

Sometimes, moving code to singletons (to minimize code duplication, for example) can also decrease cohesion: You move stuff to different classes / packages that conceptually should be a part of the original class / package.

Why is this a problem?

In a tightly coupled system, changes ripple through the system: You want to change one tiny bit of functionality, but to do that, you'll have to modify 20 classes and 50 tests.

In a system with low cohesion, it is hard to even find out where to make a change: It can be any one of 10 different modules. Or it could be all of them. Sometimes it's even hard to know if you have found all the places you have to change to fix one tiny defect.

One and Only One Object

If you really want only one instance of a certain class, implement inversion of control for your dependencies. If you are already using a dependency injection container (like Guice or Spring), just make sure all your singleton objects are managed by that container. All those containers have a way to declare objects as singleton.

If you don't want to use one of those containers, just create all instances of objects you'll need to be singleton in one place, and inject them to other objects via their constructor. You may only need one instance throughout the lifetime of your application, but you don't need the Singleton pattern for that. Just construct one, and inject it into your classes as desired.

And if you don't have inversion of control for your dependencies right now, start using it. It will pay off.

Stateful vs. Stateless Singletons

Do you remember the "//possible some instance state" in "InterestingThingsDoer" from above? There are basically two types of singletons:

  • Stateless singletons: You don't need more than one instance of this class, because it has no state. But you could have many of them: It just does not matter. Because they have no state :)
  • Stateful singletons: You must have exactly one instance of this class, because it represents some globally shared state.

The first category is not a problem. Just use one of the "sane" approaches for creating singletons from above, don't use the "Singleton Pattern". And even if you construct more than one, it's probably not a problem: The class is stateless anyway, you'll just end up consuming more resources. This overhead is probably negligible for most applications (but not all!).

The second category is probably even a problem when you use one of the "sane" approaches to make sure there is only one object: Shared, mutable state makes it easy to introduce defects when changing stuff and makes it hard to scale applications.

Thanks to Samir Talwar who convinced me to write this blog post. He also helped me review the very first draft of this post.
Want to read more like this? Never miss one of my articles: Readers of my newsletter get my articles before anyone else. Subscribe here!

Posting Type: 

Agile Software Development Workshop

Last week, I hosted a training/workshop about "Agile Software Development" for a customer. They had already started introducing Scrum in their teams. And actually they had already gotten quite far: They had all of the basic things in place (the meetings, task board, user stories, ...).

I did agile training in the past, but this time I did some things differently: I basically had next to no slides, and I let the audience do most of the work. We tried to learn everything together, discussing and drawing on flip charts. Also, this was the first agile training I did where all the participants were from the same company.

Well Prepared...
Well Prepared...
Live Drawing
Live Drawing

Break almost over...
Break almost over...

Here's what went well, and things that positively surprised me:

  • I thought we would do the training in a meeting room at the customer's office. But no: They rented a really nice villa for two days!
  • They brought in almost everyone from three different teams, plus managers, project managers, ... Basically everybody who has to do with software development.
  • They already tried to implement Scrum on their own (quite successfully). So everyone had at least a basic understanding of "Agile" and we were able to spend a lot of time discussing (and trying to solve) their concrete problems/challenges. Because of that, the workshop was very hands-on.
  • Everybody participated in all the exercises.
  • In some of the exercises, I was really just a facilitator. The participants researched, discussed, drew the flipchart pages, ...
  • The weather was really nice (especially for February), and we were allowed to also use the garden of the villa, so we did some of the sessions outdoor.
  • I had to shorten some topics on my agenda, but I actually expected that. And I did that after a discussion with the participants, so they got to decide what they wanted to do in-depth and what we could shorten.

I took pictures of all the flip chart pages and sent them everything I drew on the whiteboard. I also sent them lots of links and books for further reading, grouped by topic. So, even though I didn't really have any "course material", I hope the participants have enough information to remember what we discussed/learned anyway.

Outdoor Sessions
Outdoor Sessions
Group Discussion
Group Discussion

Preparing to discuss
Preparing to discuss
Writing good user stories
Writing good user stories

Again, the best thing about this particular workshop was, that they had already tried some things, and had a lot of concrete questions that we could answer together. We also tried to solve some of their problems together, and worked out a list of things for them to try next.

There were also some things that didn't go so well, and things I need to consider for future workshops/trainings:

  • The time was almost too short - For that group size and what they wanted to learn, three days would probably have been better than too.
  • The days were almost too long - But we took a lot of breaks, so I guess it was OK. At least, nobody fell asleep.
  • Some people told me afterwards that they really would have wanted more slides and a hand-out. I guess I will prepare some for future workshops, but also inform the participants in advance that this is mostly a workshop, not a talk or training with ex-cathedra teaching.
  • The group size was rather large, and during my planning, I didn't really consider that some activities are quite slow with larger groups. Had the group been even bigger, we probably would have needed a third workshop day or cut even more topics from the agenda. I have to consider this when planning future trainings / workshops.

For me, it was a great experience. And the first results from the anonymous survey I sent to the participants are also very positive. In future trainings or workshops, I will definitely use a similar teaching style, but change a thing or two to address participant concerns from this workshop and earlier trainings I gave.

Facilitated discourse
Facilitated discourse

Preparing for a retrospective
Preparing for a retrospective:
"What did we learn?"
Results Gallery
Results Gallery

And I will have another chance to teach soon: Next month, I'll co-host a workshop about "Test Driven Development for iOS Apps" together with Rene Pirringer. Interested? You can find out more Information here (German): Mathema Workshops, and you can also register at that page. I heard that there are still some places left...

Never miss one of my articles: Readers of my newsletter get my articles before anyone else. Subscribe here!

Posting Type: 

jObjectFormatter - A toString Generator for Java

Over Christmas, I realeased a new open source project: jObjectFormatter, a simple toString() generator for java. Using it is really easy: Just override your toString methods with:

public void toString() {
    return ObjectFormatter.format(this);

Ok, there is a little bit more to it: jObjectFormatter is fully configurable. You can define how your objects should be formatted and which properties should be part of the output. You can find more information in the official documentation on GitHub:

I hope you have fun with it, and I would really appreciate your feedback.

Posting Type: 

Well Crafted Code Will Ship Faster

well-crafted code is expensive. good developers spending hours on fixing bad code is very, very, very, very expensive.

Christoph Neuroth

Christoph wrote this some days ago on Twitter. And it sparked quite some discussion. So, a few days later, Chris tweeted:

Sigh so many reactions to that tweet that read "but we need to ship". People still don't get it.

Well crafted code *will* ship faster. Today, tomorrow and in five years. Crap you ship today will kill you in the future.

Christoph Neuroth

Now, you know, the thing is:

Well Crafted Code Will Ship Faster

Actually, this has been demonstrated a couple of times in the past. As a starting point, read the chapters about quality in Code Complete:

The general principle of software quality is: Improving quality reduces development cost.
You don’t have to choose between quality, cost and time-they all go hand in hand.

Steve McConnell, Code Complete 2

Yes, writing code well takes time. But fixing defects takes a lot of time too. Defects are rediculously expensive! And most of the customers I worked with did not even think about the true cost of their defects (in terms of lost developer productivity, lost user productivity, lost credability, ops team cost, lost business, lost market opportunities by schedule slips, budget uncertainty because of bad predictability, ...). Mose companies I worked with so far simply didn't know how much money each defect cost them!

Techniques to improve quality are not always cheap. But, overall, they are free, because:

Projects with low defect potentials and high defect removal efficiency also have the shortest schedules, lowest costs and best customer satisfaction levels.

Steve McConnell, Code Complete 2

So, in software development, you cannot trade quality for speed or cost. Teams that produce higher quality also work faster and cheaper. A study about test driven development by Microsoft research also agrees: Here the teams using TDD took a little longer than teams not using TDD (estimated by Management), but:

From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM.

Again, Steve McConnell agrees too:

Writing test cases before the code takes the same amount of time as writing test cases after the code, but it shortens defect-detection-debug-correction cycles.

Steve McConnell, Code Complete 2

What To No Next?

So, you need to improve quality to go faster. The "obvious" idea for most would then be: "Let's invest in quality, then we will become faster over time". But I think this is an extremely difficult route to go. Where do you start? Where do you stop? I think, if you start with "let's improve quality", there is a real danger that this effort will lead you to gold plating and no real improvements.

I think the better way to achieve this is: Just try to move faster, and fix all the problems you face. If you ship your code to customers once a year now, try to ship every three months. When this works, ship every month. And so on. Then you know what "good enough" is at every step of improvement - You know when to stop. And this is extremely valuable knowledge.

I have seen this approach working, but I cannot fully explain why (yet). I think it has some similar reasons like First do things right, then do the right things.

Get Help

Your company and your teams can probably fix most of their problems by themselves. But sometimes it's just cheaper and more efficient to get an external expert. And sometimes you cannot solve your problems all by yourself, because of company politics. Or because you don't even see some of the problems or solutions - you developed blind spots over the years.

Maybe, right now, you think: "Yeah, he only wants to sell his services. Why should I believe him?". But so many teams would benefit from coaching by someone external to the team, especially when it comes to shipping faster. Just consider it, please.

Still not sure? Start with a question: Go to and ask me anything - I'll do my best to answer it, and it's free!

Update: After a very interesting discussion on Twitter, I wrote a follow-up where I explain all concepts and terms I used in this article and provide more reasoning why I think "better is faster": Well Crafted Code, Quality, Speed and Budget.

Never miss one of my articles: Readers of my newsletter get my articles before anyone else. Subscribe here!

You might be also interested in:

Posting Type: 

Test-driven development in Ruby

Last week I created a new contact form for this page where you can ask me anything, and I'll do my best to answer your question. From now on, I will post the most interesting questions and my answers here on my blog (and, of course, on my newsletter).

So, here is the very first question somebody asked me, and my answer. After reading it, try it yourself: Just press the big blue button at the top of this page, and ask a question...

I'd like to know more about TDD in Ruby, but most of the tutorials that I have seen have been focused on Java. Where do I start learning TDD in Ruby?

LJ Nissen

Dear LJ,

thank you for your question. It was actually the first question that was submitted through my new contact form ;) And it's already a tough one... At least for me. But I like that.

Alright. Before we start, let me say that I am by no means a Ruby expert. So take everything I say about Ruby with a grain of salt. But I don't think the choice of Language is really that important here. Let me explain...

I know that quite a lot of TDD literature is written for Java. But there are people who do TDD in Ruby - after all, the whole "Is TDD Dead?" [1] discussion got started after David Heinemeier Hansson wrote an angry blog post [2]. So, can you use Java tutorials or a book like "Growing Object-Oriented Software, Guided by Tests" [3] even though you are using Ruby? I think so. But more on that later.

Before you start with those tutorials, you need to learn your test framework reasonably well, so you're able to translate the examples from Java to your test framework. It probably also helps when you can read at least *some* Java code, but many tutorials describe the reasoning behind the code well enough so you should be able to implement it in Ruby.

There seem to be *many* different testing frameworks for Ruby [4], and I can't really help you with choosing one. But the testing frameworks seem to have some good documentation (for example, here [5]).

Now you can start to learn the TDD flow. Don't try anything fancy yet. Take a really simple example (like the "Prime Numbers Kata" [6] - The solution has less than 10 lines of code!) and write your first test. Get it to green. Refactor. Write the next test. Get it to green. Refactor.

Then you can try a fancier example. I like "Hangman" as an exercise: Write a hangman game and/or a bot that can play hangman. You can start either with the game or with the bot. The bot can be very simple (just guesses random letters), and you can evolve it to use more elaborate strategies (dictionary based, ...).

Hangman is also great because you can do it "Inside-Out" or "Outside-In" [7], and now it is probably time to learn about the differences between those strategies. Also, try to find out how you can use Mocks [8] in Ruby, because you'll probably need them for "Outside-In".

When you have a good feeling for your testing framework, mocking framework and how you can refactor code in Ruby, you can start to do more tutorials or start reading "Growing Object-Oriented Software, Guided by Tests" [3]. Even though most of the examples you'll find will be in Java or C#, by now you should be able to translate the *learnings* and *reasonging* behind the examples to Ruby.

You should pay special attention to the "refactoring" step in TDD. This is really important - If you don't do it, your software design will suffer. Also make sure that you proceed in small, safe steps. "Taking Baby Steps" [9] and "Brutal Refactoring Game" [10] are two exercises that can help you here. You normally do those exercises in a group, at a conference or during a workshop. But you can practice them alone too.

Also, after the first few exercises and Katas, try to use TDD in a "real" (i.e. longer) project - Either a side project or at work. A Kata is nice to get started, but they are quite short lived. You do them, and when you are finished, you throw away the code. You will learn different things when you have to live with your code and your decisions for a longer time, like in a "real" project.

What questions do you still have? What will you try next?

Kind Regards,


Posting Type: 

Integrated Tests - The Other Side Of The Scam

At GeeCon Prague 2015, I saw a great talk by J.B. Rainsberger about integrated tests. It was basically a live version of his essay Integrated Tests Are a Scam, that I had read earlier. But It was great to hear his reasoning live, in a narrated way.

I use the term integrated test to mean any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior.

J.B. Rainsberger

The Scam: Original Version

The gist of his talk (and his essay, which you should read), is: If you start to depend on integrated tests, you need to write more and more of them over time. Since they don't put as much positive pressure on your design as unit tests would do, your code quality will suffer, more defects will escape your net of tests, and you'll need more integrated tests to reproduce them.

The driving force here is: Positive design pressure caused by unit tests. We are missing this positive pressure with integrated tests, and this brings us into a tail spin.

The Problem with that Version

Now, some people seem to have a hard time to understand this narrative. After the talk, I have heard from several people things like:

"Integrated tests might not be the best possible solution, but they are not so bad. Better have some tests than none."

"This sounds nice in theory, but I don't think his solution (unit tests with mocks) will work in the real world. It is not any better than having the integrated tests."

"But we have to test if all components are working together, don't we?"

Some Don't Feel the Pressure

I almost dismissed what they said as "some people always complain". But then I started thinking. For me, the reasoning in the talk was absolutely plausible and the conclusion to use unit tests and mocks seemed just right. Why was this not the case for other people?

And I think I found a possible explanation...

Some people don't feel the positive design pressure from their tests. They write tests, but don't take too much care to write good unit tests. Often, they don't refactor enough. Then, after a very small change, 20 tests break, and they complain that testing is waste. I have seen this scenario, and I did those things myself.

In other words: TDD is hard. You have to learn it and train it and take the time to do it right. But when you do it right, you can get great benefits from it.

Alternative Version of the Scam

Even if you do TDD like that, integrated tests are still a scam: They still lead you down this vicious circle where you need more of them the more you have.

  1. We already have some integrated tests
  2. They don't catch many errors (because you never have enough integrated tests)
  3. When an error occurs, we try to write a test
  4. Since we already have some integrated tests, and we do not refactor enough, some parts of our system are really hard to test
  5. So, let's just add some code to an existing test or write another integrated test, and then fix the bug
  6. We have less time to write unit tests or improve our design
  7. Go to 1

I know that one can explain this cycle with "the positive design pressure is missing". But it feels differently for the people in the cycle, who are not "listening to their tests" anyway. Because if they would, they would not be in this cycle.

Second Vicious Circle

There is a second vicious circle that often seems to happen at the same time:

  1. We already have some integrated tests
  2. We don't want to cover a line of code with different tests, because we think this would be waste
  3. So we don't write unit tests, since we already covered the code with integrated tests ("There is already a test for this class, why do you write another one?")
  4. Oh shit, writing those integrated tests is hard
  5. Also, they never catch regressions (see above)
  6. So we write less of them, and make them bigger
  7. Bigger tests cover more lines -> Go to 1

Trust in Automated Testing Suffers

So, writing integrated tests is hard. And those tests seldom catch real regressions. I mean, they do, but they'll also often fail when we make a perfectly legitimate change. So, over time, people will start to consider automated testing a waste.

Also, integrated tests often become flaky. There are a simply too many reasons why they might fail, so they'll fail "for no reason" from time to time. You'll hear your team members say: "Yeah, the nightly build is red, but whatever. This test simply fails once or twice a month."

Over time, the team trusts the test suite less and less. You can easily get into a situation where a large percentage of the builds fail because of flaky tests, and nobody fixes them, because fixing those tests is hard, and they never catch any regressions anyway.

Manual Testing

When trust in automated testing decreases, teams often rely more and more on manual testing. "Red-Green-Refactor" becomes (if we ever had it) "Change-Test-Debug". They'll have testers (on the team or in an external testing department) who will "validate" the results once the programmers are "finished". At best, they will automate some test cases "through the GUI", but often they will just click through the program.

This manual testing and automating through the UI slows down all feedback cycles considerably. Later feedback means that fixing the problems we find becomes more expensive, because someone has to go back and change something that was supposed to be "finished".


You cannot write all the integrated tests you need to verify that your software works correctly under all circumstances. Even covering all the code paths is incredibly hard. So, if you rely too much on them, you will increase the likelihood that some defects will slip all your safety nets and "escape" to production.

Integrated tests are self-replicating: When you have more, you'll need them even more. Then they often become flaky and trust in your test suite starts to suffer. But when you rely too much on manual testing as a result, you slow down your feedback cycles, making your development unresponsive and expensive.

So: Be wary of integrated tests!

Now subscribe to my newsletter so you don't miss Part 2: "The Mock Objects Trap"!

You might be also interested in:

Posting Type: 

My New Playground Project: "exampilistic"

Never miss one of my articles: Readers of my newsletter get my articles before anyone else. Subscribe here!

So I am working on "exampilistic" again... Some time ago (actually a looooong time), I wrote about some some ideas how one could improve "Specification by Example tools (especially FitNesse) and how I would implement such a tool. Even though I had working code back then, my approach didn't really work the way I wanted it to. So I put my code on Github and stopped working on it.

Now I am playing around with those ideas again, but I want to do some things quite differently. In this post, I want to tell you about the design goals for the new tool and what I want to do differently this time.

Side Note: I think that "Specification by Example" (or Behavior Driven Development or Agile Acceptance Testing) is a really useful and important technique. If you want to know more, try one of the existing tools, like FitNesse or Contact Me.

Major Design Goals

  • Wiki. I like the "Wiki" part of FitNesse - This works very well for me. I can click through the documentation and move stuff around. I can create living documentation that everyone can read, browse and edit - Even non-technical users. I think the reading and browsing part is harder with tools like SpecFlow or Cucumber, where your specifications are in source files...
  • Separation of tests and pages. I don't like the fact that in FitNesse, a page is a test. I want a page to include one or more tests. This will enable me to move tests around and link them from multiple pages. First, the test is only linked from a user story. Then, when the functionality is complete, we write the documentation and link the test there too...
  • Generators for the glue code. With FitNesse (and some other BDD / Specificaiton by Example tools), the tests and the glue code can easily get out of sync. So I want to generate all the glue code. This will be quite hard, because the generators need some refactoring capability.
  • No test runner. FitNess is a test runner. You can click a button, and it will run all your tests. In exampilistic, you will not be able to do this. It will generate language specific test code for you: For Java, it will generate JUnit tests, for Clojure it will generate clojure.test tests, and so on.
  • Specialized editors for the tests. The table syntax in FitNesse is a bit hard to get right. As is the format for cucumber. SpecFlow and Jnario, OTOH, have great editors for the tests. So, exampilistic will also have specialized editors for different kinds of tests, that support you to get the syntax right.
  • Test refactoring support. This is related to the generators: When someone does a minor change to a test (like, re-naming a column), it should be possible to just generate the glue code and run the test again, without intervention from a developer. Many current tools don't support this.

Minor Design Goals

  • Clojure / ClojureScript. Actually, when I started again, I didn't want to create a new BDD tool at first. I only wanted to get better with Clojure and ClojureScript. And I wanted to try Re-Frame. But I don't really like Code Katas or example projects - I always have the feeling that I don't learn much from them. So I decided to work on a larger project for some time, and "exampilistic" came to my mind.
  • Markdown. Well, everyone knows it. And it's easy to learn. And I found a markdown parser library that's easy to use from Clojure and ClojureScript.
  • Interactivity. For the specialized editors, I need a lot of interactivity. During my first try (some years ago), I was using AJAX for this. That didn't work too well. Now I want to try with a "Single Page Application" using Re-Frame and React.
  • Search engine optimization. Despite the interactivity, I want to be able to render static pages on the server using the same code I use on the client. For search engine optimization and because it is a nice challenge (That I have already solved).

Differences to Last Time

  • Clear separation of responsibilities. In my last attempt, I tried to do too much in the Wiki itself. It would be a test runner (like FitNesse). And I would store all content as Java Classes (to ensure the test code is always in sync with the wiki). This time I want to do it differently: The wiki is just there for the test definition. The glue code will be generated by a separate code generator. And the test runner is whatever test runner you use in your language (e.g. JUnit when you use Java).
  • Probably no automatic feedback from code to wiki. In my last attemt, when you would refactor the code (e.g. rename a class), you would also see that change in the Wiki (because the wiki stored all pages as java classes). I don't think this is really needed. And if I need it, I will create a separate tool that will feed back information from the code to the Wiki. Because: Clear separation of responsibilities.


"exampilistic" is, so far, only a toy project. But so far it also looks very promising. My progress is better than I would have expected. I also think I learned a lot from my first attempt, and this time, the design goals are mostly sound (But I'm sure some of them will change when I learn more during developing functionality).

It is still to early to show you some code or screenshots, but if I manage to get to a point where some core functionality (like editing a single wiki page) is fully usable, I will make the project open source. And also show you some screenshots.

Are you interested in managing IT and software projects or teams? Are you in Europe in Autumn 2016? We are preparing a great conference for you: Advance IT Conference

Posting Type: 

Clojure "CompilerException: java.lang.Exception: Nothing specified to load, compiling: [file name]"

Never miss one of my articles: Readers of my newsletter get my articles before anyone else. Subscribe here!

Today I had a strange exception when compiling some crossover code for Clojure and ClojureScript:

Caused by: java.lang.Exception: Nothing specified to load
Reloading Clojure file "/[path]/[file].cljc" failed.
clojure.lang.Compiler$CompilerException: java.lang.Exception: Nothing specified to load, compiling:([other path]/[other file].cljc:1:1)

There was no further information in the error message, and I didn't find out anything useful by googling. During my trial and error (with previous versions of the code), I found out that the problem was in:

(ns reframe.serverside
    (:require #?(:cljs [re-frame.core :refer [dispatch-sync]])))

When compiling this code to ClojureScript, everything works find, because the "require" conatins an entry: "re-frame.core". But when this file is compiled to Clojure, the "require" is empty, and the compiler does not seem to like this. Here is the fixed code:

(ns reframe.serverside
    #?(:cljs (:require [re-frame.core :refer [dispatch-sync]])))

I just remove the whole ":require" when compiling to Clojure. (Anyway, a better error message would have been really useful here...)

Are you interested in managing IT and software projects or teams? Are you in Europe in Autumn 2016? We are preparing a great conference for you: Advance IT Conference

Posting Type: 

Server-Side and Client-Side Rendering Using the Same Code With Re-Frame

Never miss one of my articles: Readers of my newsletter get my articles before anyone else. Subscribe here!

In the last few days, I was playing around with Re-Frame. Check it out, it is awesome. Even if you're not planning to ever use clojure - Even if you're not even doing web development - At least read the It's the best I've ever read!

Anyway, I was playing around with Re-Frame. And after I got the simple example running, I tried to make it do something it's not supposed to do: Render a page on the server side, using the same source code as on the client side. If that works, it would be great: You could create a single page application that renders everything on the client, but still provide basic functionality when JavaScript is switched off. And it would be great for SEO: Search engines get plain HTML and don't have to interpret JavaScript to see the content.

The things I tried basically work now, but it's not 100% usable yet. Here's my current solution, and what already works...

I will not explain how re-frame or ClojureScript work in this post, so you probably should read the re-frame before continuing...

Simple Example Rendered on the Server

So, I started with the simple Re-Frame example and tried to render the page on the server.

With Clojure 1.7's Reader Conditionals I can now write code that can run on the server (as Clojure code on the JVM) and also on the client (ClojureScript compiled to JavaScript). Basically, everything that's in a ".cljc" file is available on the server and on the client, while ".clj" is only available on the server and ".cljs" is only available on the client.

I created the following project structure:

    +---clojure       <-- Clojure code to run on the server (.clj)
    +---clojurescript <-- ClojureScript code to run on the client (.cljs)
    \---crossover     <-- Crossover code to run on server and client (.cljc)

Then I simply copied the ClojureScript code from the simple example to src/main/clojurescript and created a hiccup view in src/main/clojure that created a page that looks exactly like the HTML page from the simple example. After everything was running (which required a tweak or two), I started to move stuff around.

Crossover Code

I moved all the "view" functions from the ClojureScript file to a new crossover file called "renderer.cljc". The clojurescript then should only call one function from the renderer. The blue code is unchanged, I have only changed the bold lines and removed all the "View Components":


(ns simpleexample.core (:require-macros [reagent.ratom :refer [reaction]]) (:require [reagent.core :as reagent] [re-frame.core :refer [register-handler path register-sub dispatch dispatch-sync subscribe]]
[ :as renderer]))
(def initial-state {:timer (js/Date.) :time-color "#f34"}) (defonce time-updater (js/setInterval #(dispatch [:timer (js/Date.)]) 1000)) ;; -- Event Handlers ---------------------------------------------------------- (register-handler ;; setup initial state :initialize ;; usage: (submit [:initialize]) (fn [db _] (merge db initial-state))) ;; what it returns becomes the new state (register-handler :time-color ;; usage: (submit [:time-color 34562]) (path [:time-color]) ;; this is middleware (fn [time-color [_ value]] ;; path middleware adjusts the first parameter value)) (register-handler :timer (fn ;; the first item in the second argument is :timer the second is the ;; new value [db [_ value]] (assoc db :timer value))) ;; return the new version of db ;; -- Subscription Handlers --------------------------------------------------- (register-sub :timer (fn [db _] ;; db is the app-db atom (reaction (:timer @db)))) ;; wrap the computation in a reaction (register-sub :time-color (fn [db _] (reaction (:time-color @db)))) ;; -- Entry Point ------------------------------------------------------------- (defn ^:export run [] (dispatch-sync [:initialize])
(reagent/render [renderer/simple-example]
(js/document.getElementById "app")))

Now, obviously, this didn't work out of the box. The crossover code (in the .cljc files) was missing all of the Re-Frame dependencies. Also, the hiccup collection used by Re-Frame is slightly different than what hiccup expects on the server (since Re-Frame has to process the template to handle subscriptions and other things). And, third, the ClojureScript event handler for the input field obviously doesn't work when rendering on the server.

I could provide the missing dependencies for ClojureScript with a reader conditional, but I'd also have to provide an alternative implementation when compiling as Clojure code. Fortunately, this is quite easy. Then I created a function that translates from a Re-Frame hiccup collection to a regular hiccup collection, that is only used on the server (but I need it only in the server-side hiccup view - see later). And third, I decided to use an empty event handler function when rendering on the server.


;; Import Re-Frame dependencies when compiling as ClojureScript
  #?(:cljs (:require [re-frame.core :refer [dispatch-sync subscribe]])))

;; Use this app-db instead of the Re-Frame app-db when compiling as Clojure
#?(:clj (def app-db
          (atom {:time-color (atom "#77f")
                 :timer (atom "--.--.----")})))

;; Use this Re-Frame's "subscribe" when compiling as Clojure
#?(:clj (defn- subscribe [v]
          (get @app-db (first v))))

;; -- View Components ---------------------------------------------------------
(defn greeting
  [:h1 message])

(defn clock
  (let [time-color (subscribe [:time-color])
        timer (subscribe [:timer])]

    (fn clock-render
      (let [time-str (str @timer)
            style {:style {:color @time-color}}]
        [:div.example-clock style time-str]))))

(defn color-input
  (let [time-color (subscribe [:time-color])]

    (fn color-input-render
       "Time color: "
       [:input {:type "text"
                :value @time-color
                ;; Use empty on-change handler when compiling to Clojure
                :on-change #?(:clj  ""
                              :cljs #(dispatch-sync
                                       [:time-color (-> % .-target .-value)]))}]])))

(defn simple-example
   [greeting "Hello world, it is now"]

And here's how I translate (simple-example) on the server to create the static content:


     [page :refer [html5]]
     [element :refer [javascript-tag]]
     [page :refer [include-js]]]
    [ :as renderer]))

;; Create a regular hiccup collection from a Re-Frame hiccup collection
(declare reframe>hiccup)

(defn- map-reframe>hiccup
  (cond (keyword? reframe-hiccup-element) reframe-hiccup-element
        (map? reframe-hiccup-element) reframe-hiccup-element
        (vector? reframe-hiccup-element) (if (fn? (first reframe-hiccup-element))
                       (let [result (apply (first reframe-hiccup-element) (rest reframe-hiccup-element))]
                         (if (fn? result)
                           (apply result [])
                       (reframe>hiccup reframe-hiccup-element))
        :else (throw (IllegalStateException. (str "Illegal element " reframe-hiccup-element)))))

(defn- reframe>hiccup
  (into []
        (map map-reframe>hiccup reframe-elements)))

;; create the hiccup collections that represent the static page
(defn show-page [page-name]
     [:title "Replace me"]]
     [:div {:id :app}
      (reframe>hiccup (renderer/simple-example))]
     (include-js "/js/goog/base.js")
     (include-js "/js/main/main.js")
     (javascript-tag "goog.require(\"simpleexample.core\"); window.onload = function () {; }")]))


Creating the static HTML on the server is already working. Just uncomment "window.onload = ..." in show-page and you'll still see the correct content on the client - But without the dynamic behavior. And the server side rendering uses (almost) exactly the same code as the client side rendering - So we can be sure to see (almost) the same page as if the client would have rendered it - Except for the date, which will only be rendered by the client.

What's still missing is that the server should make sure that the client uses the same values in it's app-db as the server used for rendering the static content. I think this is not that hard, I just didn't implement it yet.

Another inconvenience is that the server will render the whole page again in window.onload, throwing away anything the server rendered. I was thinking, maybe I could only re-render everything on the first user interaction. I.e., only when the user clicks a link, start Re-Frame and render the new state...

Maybe I'll package up everything as a small library when I get it working. I could also create a pull request for Re-Frame, but right now I think a standalone library (e.g. Re-Frame-Server) would be better...

Feedback Welcome

What do you think about this solution? Do you think I can solve the remaining problems? Will it be usable in bigger applications, or will I hit a wall somewhere? Do you have any questions? Please Contact Me!

Are you interested in managing IT and software projects or teams? Are you in Europe in Autumn 2016? We are preparing a great conference for you: Advance IT Conference

Posting Type: 


Connect with David Tanzer: Send me an email:
A curated list of posts with summary information can be found here: My Favorite Posts.

Subscribe to RSS - Professional