David's Blog

REPL Driven Development and Testing in Clojure

Note: This is a really simple example. But I'm quite confident that this could work on bigger problems / projects too.

On the train home from SoCraTes 2014, I was discussing with Johannes Link if REPL driven development could give us the same advantages we get from Test Driven Development (TDD).

We agreed that TDD gives us at least these three advantages:

  • Fast Feedback: With a good unit test suite, you quickly get feedback about what is going on in your code base. Especially with a plugin like Infinitest
  • Small, safe steps: By writing the tests first, you automatically validate every step. By writing good tests, you force yourself to take small steps.
  • The tests: They are a living documentation of our thinking process and a regression suite for our code base.

That a REPL gives you fast feedback is obvious. But what about the other two?

Small, safe steps

We quickly agreed that the REPL allows you to take small, safe steps. You change a little bit of your code, then you immediately try it out. Let me show you with an example:

In this example I want to print all prime numbers up to a given number. For this I'll implement a lazy sequence of prime numbers using the Sieve of Eratosthenes [1]. I'll first solve the problem directly in the REPL, then I'll copy the solution to a source file.

The sieve initially works on a list of all positive integers, starting at 2. So we'll define this first.

(take 10 (iterate inc 2))
=> (2 3 4 5 6 7 8 9 10 11)
(defn to-sieve [] (iterate inc 2))
=> (var user/to-sieve)
(take 10 (to-sieve))
=> (2 3 4 5 6 7 8 9 10 11)

Now, when we know a prime, we have to remove ("cross out") all it's multiples from the list of numbers. We'll use "filter" for this.

(defn remove-multiples [prime s]
  (filter #(not= 0 (mod % prime)) s))
=> (var user/remove-multiples)
(take 10
      (remove-multiples 2 (to-sieve)))
=> (3 5 7 9 11 13 15 17 19 21)
(take 10
      (remove-multiples 3 (to-sieve)))
=> (2 4 5 7 8 10 11 13 14 16)

We know that the first number in the list is a prime number. So we can define a lazy sequence "lazy-sieve" that takes a sequence of numbers as a parameter. Based on it we can define a function "sieve" that provides the list [2].:

(defn lazy-sieve [s]
  (let [prime (first s)]
    (cons prime
          (lazy-seq
            (lazy-sieve (remove-multiples prime (rest s)))))))
=> (var user/lazy-sieve)
(take 10 (lazy-sieve (to-sieve)))
=> (2 3 5 7 11 13 17 19 23 29)
(defn sieve []
  (lazy-sieve (to-sieve)))
=> (var user/sieve)
(take 10 (sieve))
=> (2 3 5 7 11 13 17 19 23 29)

The lazy sieve removes the first number from the sequence of numbers - We know this must be a prime number. It combines this prime number with the lazy sequence of other prime number ("(cons prime (lazy-seq ...))"). To create the lazy sequence, it simply removes all multiples of the current prime from the rest of the sequence and passes it to "lazy-sieve" again.

Now we can define "primes-up-to n", which takes prime numbers from this lazy sequence as long as the current prime nubmer is less than n:

(defn primes-up-to [n]
  (take-while (partial > n) (sieve)))
=> (var user/primes-up-to)
(primes-up-to 10)
=> (2 3 5 7)

Extracting the code

Now I can simply extract all functions I still need from the REPL to a clojure source file. This just copy/paste: Really simple.

primes.clj

(ns primes)

(defn to-sieve [] (iterate inc 2))

(defn remove-multiples [prime s]
  (filter #(not= 0 (mod % prime)) s))

(defn lazy-sieve [s]
  (let [prime (first s)]
    (cons prime
          (lazy-seq
            (lazy-sieve (remove-multiples prime (rest s)))))))

(defn sieve []
  (lazy-sieve (to-sieve)))

(defn primes-up-to [n]
  (take-while (partial > n) (sieve)))

But what about the tests

On the train, Johannes and I were discussing how you could come up with tests after everything already works. I was like "Well, you could think about what exactly you want from a regression test suite and... Wait a minute, the tests are already there! You already have input/output pairs in the REPL that you validated as correct! Just generate the tests from them!"

Extracting automated tests

So now we can generate automated tests from the things we did in the REPL. Everytime I was trying a function with concrete parameters, the REPL told me the result, and I manually checked the result for correctness. So now I only have to find those results that were correct. I can use these to generate automated test cases. I only have to reorder the function call / result pair and wrap it in a call to "expect" and voila: I have a test.

So, this REPL output would result in the following test:

(take 10 (to-sieve))
=> (2 3 4 5 6 7 8 9 10 11)

(expect '(2 3 4 5 6 7 8 9 10 11)
        (take 10 (to-sieve)))

This is a little bit more work than copy/paste, but not much more. Copy the function call and the result to a file, reorder, wrap them in "expect". After extracting all the tests from the REPL, the resulting clojure file looks like this:

primes_test.clj

(ns primes-test
  (:use [primes]
        [expectations]))

(expect '(2 3 4 5 6 7 8 9 10 11)
        (take 10 (to-sieve)))

(expect '(3 5 7 9 11 13 15 17 19 21)
        (take 10
              (remove-multiples 2 (to-sieve))))

(expect '(2 4 5 7 8 10 11 13 14 16)
        (take 10
              (remove-multiples 3 (to-sieve))))

(expect '(2 3 5 7 11 13 17 19 23 29)
        (take 10 (lazy-sieve (to-sieve))))

(expect '(2 3 5 7 11 13 17 19 23 29)
        (take 10 (sieve)))

(expect '(2 3 5 7)
        (primes-up-to 10))

(run-all-tests)

Where running the last line results in the output:

Ran 6 tests containing 6 assertions in 14 msecs
0 failures, 0 errors.

Conclusion

Johannes then said "It would be nice if the IDE or the REPL could do this for you." Indeed it would be great if you could generate test cases from a REPL input/output pair by just pressing a button. But as long as this functionality is not there, generating them by hand is not that hard.

For me this means that I can get all the advantages of TDD by using the REPL - At least with clojure. Maybe this might be a bit harder with other languages - I did not try it. Yet.

[1] Whether or not this is a good solution is another topic...
[2] I know I could have combined the two. To me, separating them seemd clearer for now.

You might be also interested in:

  • Simplicity: Simplify. Then simplify more. And: Don't let the code become complicated in the first place.
  • Cheap plastic drills: Most people think construction workers should have great tools. A lot of people think paying more than 1000 Euros for an office chair is a waste of money, even for a software developer who uses it 8 hours a day. Good tools are expensive.
  • Mocks or Intermediate Results: What I Would Do: An answer to Kent Beck's article, where he wrote about how he uses intermediate results instead of mocks. I show an alternative approach.
  • Improve your Agile Practices: A FREE course that teaches you how you can improve as a software development team
Posting Type: 

New Frontpage

A few days ago, I have completely re-written the front page of my homepage. I think now it better shows what I have to offer and how I can help teams and software organizations. I would be really interested in your feedback: What do you like about it? What do you dislike? Have you any suggestions what I could do better? Please email me and let me know: Business@DavidTanzer.net.

I also have worked on some home page projects for customers, so I have created a page that shows what I can do for you - Even though it is not my main field of activity: Ihre Webseite (German only).

Posting Type: 

No True Scotsman in Agile

In a recent discussion about whether Agile is dead or not, I read an interesting comment:

I've seen this cycle a few times. Let me handwave how it goes:

  • Agile Critisism: The snake oil is all over and getting worse!
  • Agilista: It's not done properly!
  • Agile Criticism: No True Scotsman!

pnathan on Hacker News

Indeed, I too have seen this discussion happening. So, are we really using the "No True Scotsman" fallacy when we say that failed agile projects were not done properly?

We commit the "No True Scotsman" fallacy when we exclude counter examples from our theory (or statement) by modifying the theory (statement):

Person A: "No Scotsman puts sugar on his porridge."
Person B: "I am Scottish, and I put sugar on my porridge."
Person A: "Well, no true Scotsman puts sugar on his porridge."

No True Scotsman on Wikipedia

So, if I say "Any agile project that failed was not done properly", it would be true: I would be committing this fallacy. But I don't think this is the case in many of those discussions. Many of the discussions/blog posts start like this:

  • The snake oil is all over and getting worse! Look at project [X]. They used [methodology Y] and [tools A, B and C], and they failed horribly!
  • Every [methodology Y] project I did was the horror. [methodology Y] is just an excuse for managers to micro-manage.
  • ...

In that case, if I can find specific reasons why those projects do not fit the definition of agile software development, I do not commit the "No True Scotsman" fallacy. I merely reject your example, I do not modify my statement about agile or theory of agile.

Now, I know that there is no definition of agile. But there are some things that come close to a definition. Just assess the project under discussion (or your current project) against one of them.

For example, a very simple and powerful definition is "agile as a doctrin":

Reduce the distance between problems and problem-solvers: How far is your development team separated from actual users? Can they talk to them? How far are your team and the actual users separated from budget decisions? Product initiation Decisions? Have you really reduced the distance between problems and problem solvers?

Validate every step: Do all your user stories really reflect customer needs? Does the business reason in your stories really reflect customer goals? Does every sentence on the story card add value? Does no sentence on the story card constrain the technical solution? Does every part of your solution address the needs / move you toward the goals? Are your tests adding value? Are all your deployments correct? Are you really validating every step?

Take smaller steps: How many user stories are in your backlog? Could you get away with less? How often do you meet with real users? Can you do it more often? How large are your user stories? Can you make them smaller? How often do you deploy to test and live environments? Can you do it more often? How long are your budgeting / planning cycles? Can you make them shorter? Are you really taking smaller steps?

Clean up as you go: Do you always refactor you code when all the tests are running? Do you always refactor your tests? Do you delete redundant / outdated tests? Do you regroup acceptance tests by feature set after stories are done? Do you keep your backlog short and sorted? Do you re-write/improve user stories? Do you remove meetings/artifacts/procedures from your process when they do not add value? When was the last time you edited your definition of done? Are you really cleaning up as you go?

So, if you are not really reducing the distance between problems and problem solvers, if you are not really validating every step, if you are not really taking smaller steps or if you are not really cleaning up as you go, then you are not doing Agile as good as you can (according to this definition). And if your project fails because of this [1], I will point it out. (And the questions show you where you could try to improve.)

The Agile Manifesto is another definition you could assess your project against. It only contains four values and twelve principles. Just ask yourself: Are we really valuing those four things? Do we really follow those twelve principles?

If you assess projects like this, you'll probably realize that many projects that claim to be agile have a lot of room for improvement. And some of these projects don't really try to improve. So, are they really agile? Many of the "agile is snake oil" stories are in this category.

On the other hand, you might realize that some projects who don't claim to be agile are in fact doing pretty well when assessed against those definitions. Should we really count them as "not agile" in the discussions?

Maybe we really sometimes hear "No True Scotsman" arguments in the ongoing discussions about agile software development and the death of "Agile". But often, the kinds of projects being discussed arguably were not doing agile properly. When we call that out, we are not committing a "No True Scotsman" fallacy. We are just rejecting an incorrect example.

[1] Ok, I know it's hard (or even impossible) to know exactly why a project has failed. Anyway.

I have written about "doing agile properly" in the past. You might be interested in my articles Scrum... But? and ScrumBut... and the long run.

You might be also interested in:

Learn more about how I can help you save money and earn money by improving your agile software development process.

Posting Type: 

We don't need a foreman

Last week, Robert C. Martin ("Uncle Bob") wrote a blog post about how software teams need a foreman. A couple of days later, he responded to the first wave of criticism. He is wrong about this one.

First, I thought I'll let better people than me respond to those blog posts. But after some interesting discussions on twitter, I decided to write a longer post. Because, reasoning why the foreman idea is wrong takes more than 140 characters - at least for me.

Note: None of the stories in this post did happen exactly as I tell them. Some are a mix of several experiences I had with several teams, and stories good friends told me. Some are entirely fictional. So, they are not true stories, but there is truth in them. Also, they are probably a bit exaggerated.

The foreman Uncle Bob describes here is the guy who makes sure all the tests are written. The guy who makes sure that all the exceptions are caught. The guy who makes sure all the errors are checked, and that references can't be null, and that variables are thread-safe. The guy who makes sure that the programmers are pairing enough, talking enough, planning enough. He is the only one with commit rights. He will review everything the other team members produce, and reject those packages that are not sufficient in his eyes. Here ares some of the reasons why this is a terrible idea.

1. Having a foreman destroys trust, team spirit and motivation

When a team has a foreman, this is evidence that there is a severe lack of trust within the team. Obviously, most team members are not trusted to commit stuff - because they do not have commit rights. We already know that the most experienced programmer - the foreman - does not trust some others to do their job. So there is no reason for team members to trust each other.

There could be teams where the foreman trusts all team members, the team members trust each other and everybody trusts the foreman. But I doubt it will stay that way for very long. Maybe the team changes. New members will be distrusted by default. Or the foreman makes some bad calls. Trust in him will suffer. From there it will go down.

Team members will either work together against the foreman or work against each other to please the foreman. One person with commit rights is equal to individual performance assessment of team members! Everybody knows how often Fred, the foreman, has rejected commits from Sam, the system programmer. Team spirit: Gone. And with it: Motivation.

2. The foreman is not your peer

He is the one with the commit rights. He can reject everything you did, for any reason. Do I have to say more?

Actually, I have to. At some point, Fred noticed that several team members were making the same mistakes over and over again. They even would not agree with him that they made mistakes - They insisted their code was good! Fred decided to write a weekly newsletter with common mistakes, so all team members could learn something. This obviously led to a lot of finger pointing because it was easy to find out about which commit Fred was talking.

Also, Alice, the architect, a close friend of Fred, decided at some point to rewrite code so that it would please Fred. This would save Fred a lot of time. Don, the developer, would check something in, and hours later the code was completely changed. Don was not allowed to change it back, because only Fred had commit rights.

Even if the foreman was your peer when he startet, this will change. Because he will make calls you don't agree with. And you have no power in the discussion.

3. The foreman will not give up his position

He has earned it. He is the most experienced developer. Of all the other developers who have been here long enough to be considered as foreman, he is clearly the best.

Sadly, this becomes a self fulfilling prophecy. Everybody has to agree with the foreman. He might listen to your arguments, but if he still disagrees, you have no way of winning this argument. Even if the whole team agrees with you. So you either do what the foreman says, or you leave.

In such a situation, the good developers leave first. The ones how stay are the ones who have to. Do you really want to work with a team where most members have no other option?

Fred (actually, one of the Freds I know) even told me once: "Our programmers do not want more responsibilities. We have hand-picked people who only want to program and do nothing else."

4. The foreman is probably not "the best"

The real problem that arises when the foreman will not give up his position is that he is probably not the best. Not in every aspect of the project.

In any non-trivial project you need a lot of different skills. You need someone who is really good at TDD. A hibernate wizard. Somebody who can tune the DB statements. Someone experienced at object oriented design. Architecture. Performance tuning. User interfaces - Maybe even many differen UI technologies. EAI. External systems. The best person on your team in each of these areas is probably (or hopefully) not the same person.

Or, even if he is the best now, in every area of experience needed for the project, he might not stay forever. Fred (as always, one of them) was a good programmer. He really knew the technology stack. But his object oriented design was a bit old fashioned and simplistic - You know, "dog extends animal", no SRP or OCP. But you could not argue with him about OOD because he was the foreman.

5. The foreman will have the sole responsibility for quality

I mean, hey, the foreman will tell me if I have enough tests. I'll just commit.

The project has a bug? Why did the foreman not spot it?

How should I have known that performance in this part of the code is critical? We have a foreman for that!

This is not good. When quality is not the responsibility of the people who write the code, they will stop to care. And then, the code review will be there to spot low quality, not as a quick check if quality is still high. And then your process does not have quality built in. It has quality assurance afterwards. Your lead times will increase because you will find more and more issues during review that could have been prevented before. Your cycle time will decrease because you have more work in progress caused by more rework. Your progress will slow down. And quality will not be any higher - it will probably be lower too, because you can not find all the issues after the fact. You have to build quality in. This is a lose-lose-lose situation.

6. The foreman will have the sole responsibility for architecture and design

He will only commit code that fits his ideas of the architecture and design.

Alice (...one of them...) was the official architect on the team, and management trusted her decisions. If there was a longer discussion about architecture, management would always be on Alice's side. Even when all others had a different opinion. At some point, most team members gave up arguing with Alice. They just did what she told them.

Her decisions were mostly good. Maybe not the best - we don't know because we never tried something else - but good enough for the product to be a success. But sometimes she screwed up - like we all do from time to time. But since she was the only one who made architecture decisions, those situations did not end in a "we are all in this together" mood in the team. It led to schadenfreude and finger pointing.

At some point, the foreman might even decide that the reviews start to take really long, so he'll just tell the developers how to do their tasks. This will speed up the whole process (Fred (you'll never know which one) did). Since he already has an idea how to implement all the tasks, he also has the final call on estimates. I mean, he is the best and most experience developer! Of course, management trusts his estimates more than some people with poker cards!

7. The foreman is the bottleneck

All tasks have to be reviewed by him, and if he rejects them a lot, the project will be behind schedule. If he does not have enough time to review all the tasks, the project will be behind schedule. Or have an architecture or design he does not approve of *Gasp*. So he'll have to tell the developers exactly what to do - then he does not have to reject tasks so often (see above).

8. The foreman will have the sole responsibility for schedule

Because he has the sole responsibility for quality and architecture and design, he will also have the sole responsibility for the schedule of the project. Uncle Bob even agrees:

If you want to get a project done, done right, and done on time, you need a foreman.
Robert C. Martin, Where is the Foreman?

He will be responsible for the schedule because he is the bottleneck. And he was responsible for the estimations that led to the schedule in the first place. So if the project misses a deadline, this can be traced directly back to him.

Since Fred was responsible for the estimates and for the schedule, estimates were heavily padded. Sometimes, team members would sit around for days and do nothing. Because they didn't have to - There still was time.

9. The job can be done by the whole team

If one person can do the job, the whole team can:

We work in pairs, do TDD, and raise pull requests. Every one reviews and commit code. We don't want or need a "foreman".
We commit multiple times a day as well. Pull requests are small, reviewed by peers in minutes.
Sandro Mancuso, on Twitter here and here

Uncle bob answered

Good. High functioning team. What will you do when the business doubles it with noobies.
Robert C. Martin, Twitter

Ok. First, you do not call a software developer "noob". Ever. Even if they came straight from university. They have been studying for several years and are highly trained academics. They still have much to learn, but they are not "noobs".

Second, this does not matter. When management doubles the team with new people, there are still enough experienced developers to do pair reviews of other pairs with them.

So, the whole team can do code reviews. This means the whole team should do them. Then, everybody will be responsible for the quality of the product, the design and architecture of the system and the schedule of the project.

10. Maybe new developers should do the code reviews

Good object oriented code can look like severe over-engineering to an inexperienced developer. At least, it looked like this to me. Alice would re-write my code so that classes were really small and methods would only have 3-10 lines and call lots of private methods which called other private methods, all with a really long name. It was awful. Over-engineered. When I asked here why she did it, she just told me it was better this way. Also, Fred would approve of this code.

Now I know she was right. And now I know why she was right. But since discussions were fruitless anyway (see above), i just shut up and went on.

On the other hand, hopelessly over-engineered code might look OK for people who have been working with it long enough. Sam had no problems with a class hierarchy 12 layers deep.

Maybe new developers should have the right to reject code they don't understand. Maybe they should have the right to reject code that looks over-engineered to them. The following discussions within the whole team will probably bring new insights for everybody, not only for the newcomer.

11. Young developers probably learn most from their peers

Maybe Alice even told me why her way was better than mine. I don't remember anymore. It did not stick, because I just did not believe her at the time. And discussion was fruitless anyway, so we did not discuss this any further.

In the seven years of my career, I learned a lot. And I hope to learn much, much more in the coming seven years. I learned a lot because I was lucky to work with really good people. People much better than me. But I only learned from my peers. I learned from the discussions with them. I learned from arguing with them which way was better. Most of the time, they won this argument. Because they were right and they could convince me about it. But sometimes I could convince them that I was right...

I hope I'll never stop listening to developers less experienced than me. I'm sure I can learn from them and I hope they'll learn from me. And I hope I can always work with developers who are better than me. I will try to continue to learn from them.

No, we don't need a foreman

I know a lot of what I wrote sounds like very dysfunctional organizations. But dysfunctional organizations exist. Lots of them. They do not exists by pure chance. They became that way. They might be the lowest energy state for large companies. People are not perfect. Systems deteriorate.

We have to constantly fight to keep the company in a higher energy state. We don't need a foreman. The presence of a foreman might even be a catalyst for the system to deteriorate. We need more trust, team spirit and more time to learn how to do the right thing - As a team. And we need more and better peer reviews.

You might be also interested in:

Learn more about how I can help you save money and earn money by helping your team to work together effectively.

Posting Type: 

TDD like Beethoven

Can you write good code without testing or running it? Can you refactor without even compiling the code? Can you reason about code that only exists as a text document?

Earlier this week I wrote a blog post about doing TDD only in your head, without writing anything down. I called it "TDD like Mozart" after an idea J. B. Rainsberger had in a Tweet. It was a fun exercise to do, but it has some drawbacks:

  • Overly strict rules: I could not correct any errors I spotted, because I did not allow myself to change lines already written.
  • Because of the same rules, I could not remove duplication I spotted when I wrote the code down
  • The exercise can only be done alone

Note that the first two drawbacks are actually features of the exercise. You should do all the refactoring and syntax corrections in your head.

Mozart was a great composer who did extraordinary things. Beethoven was another great composer who did extraordinary things.

Enter "TDD like Beethoven"

I decided to change the rules of the exercise a little bit. The new rules should make it easier to correct small mistakes and to work on the exercise together.

In about 1800 his hearing began to deteriorate, and by the last decade of his life he was almost totally deaf. He gave up conducting and performing in public but continued to compose; many of his most admired works come from this period.
Ludwig van Beethoven

The Symphony No. 9 in D minor, Op. 125 (sometimes known simply as "the Choral"), is the final complete symphony of Ludwig van Beethoven. Completed in 1824, the symphony is one of the best-known works of the Western classical repertoire. Among critics, it is almost universally considered to be among Beethoven's greatest works, and is considered by some to be the greatest piece of music ever written.
Symphony No. 9 (Beethoven)

What is the equivalent of "being deaf" in programming? Beethoven could not hear how his work really sounded. He could not experience his work. So, in this exercise, you are not allowed to compile or run your code. But you can - and should - write it down. Reason about it. Move stuff around. Just don't compile or run.

The Rules

  • Find a small problem you want to solve in code
  • You can use all the documentation you need
  • You can use a text editor with syntax highlighting, but...
  • Do not compile or run the solution! Do not use auto-complete!

"Compose" the solution:

  • Come up with a test and write it down in the text editor. Reason about why it must fail right now.
  • Write down the code that will make the test green. Explain to yourself why it will make the test pass.
  • Double check for syntax errors, exceptions, ...
  • Refactor the code. Explain to yourself why the refactoring is better. Explain why all the tests still pass.
  • Double check for syntax errors, exceptions, ...
  • Repeat until finished.

Run and correct mistakes:

  • Compile and run
  • Change the code so it runs and compiles
  • Write down all the changes you had to make

I guess in this exercise it will be a little bit easier to get to a running program before the first compile than in "TDD like Mozart". But I don't think the exercise will be easier overall, because this raises expectations. And it will still be really hard. But at least it will be more fun to do as a pair exercise than "TDD like Mozart".

Side note: For now, I will only write down the rules because I don't have time to run the exercise right away. I will try it out later, and I'll probably record a screen cast while doing so.

What do you think of this exercise? Are you thinking about trying it? Contact me and tell me about your experiences. If you blog about this exercise, I will gladly link your blog post here and re-tweet it. All my contact details are at the bottom of the page.

Posting Type: 

TDD like Mozart

According to the popular story (backed up by family letters), the fourteen-year-old Mozart was visiting Rome, when he first heard the piece during the Wednesday service. Later that day, he wrote it down entirely from memory, returning to the Chapel that Friday to make minor corrections.
Miserere (Allegri) at Wikipedia

A couple of days ago, J. B. Rainsberger wrote on Twitter:

Potential experiment at future conference: ask experienced TDD practitioners to test-drive a problem in their head. "TDD like Mozart"
J. B. Rainsberger

I immediately thought: "Hey, that sounds interesting!". It sounds like a hard but fun challenge. So I came up with this little exercise (probably not exactly what J. B. Rainsberger was thinking of):

Rules

  • Find a small problem you want to solve in code
  • You can use all the documentation you need, but...
  • Do not write down anything while you are "composing" the code

"Compose" the solution:

  • Come up with a test entirely in your head. Think about it in enough detail so you could write all the code down and it would compile and fail. Think about why it would fail.
  • "Compose" the code that makes the test green in your head. Again, think of all the little details, like imports, syntax, exceptions, things that might go wrong, ...
  • Refactor the code in your head

Write it down from memory:

  • Close all the documentation you used during composing
  • Now, write down your solution from memory.
  • Use a simple text editor, not an IDE. Syntax highlighting is (probalby) OK, though.
  • Do not change a line that you have already written
  • When you have written a file, close it and do not look at it again
  • Do not use copy+paste
  • Compile and run

Come back on friday to make minor corrections:

  • Change the code so it runs and compiles
  • Write down all the changes you had to make

The Exercise

This is probably a very difficult exercise. But it sounds like fun, so I'll try it. I will implement a server for the "ECHO" protocol (defined in RFC862). I'll be right back and post the solution. You have to trust me that I really followed the rules. I don't expect the code to compile anyway ;) So, please wait here while I'll do the exercise...

Thanks for waiting! I did not record a screen cast, because you'd see me reading documentation and doing nothing for most of the time. I wrote down the source code from memory, without changing lines I already wrote. Here I will add some comments about my thinking process between the files. The solution is a bit over-engineered. This is because I really wanted to include refactoring steps in my thinking process.

I started with the following gradle build script (which I wrote before doing the exercise):

build.gradle

apply plugin: 'java'

repositories {
	mavenCentral()
	maven {
		url 'http://files.couchbase.com/maven2/'
	}
}
dependencies {
	testCompile 'junit:junit:4.+'
	testCompile 'org.mockito:mockito-core:1.9.5'
}

First, let's start with an acceptance test. I thought about this test first. It describes the desired functionality from a user's perspecive. This test exercises the whole system. This test is there to make sure the system contains the desired functionality, not to drive the design or make sure it is implemented correctly. I will think of unit tests for those purposes later.

EchoServiceTest.java

import org.junit.*;

public class EchoServiceTest {
    @Test
    public void sendsBackLineOfText() {
        EchoService echoService = new EchoService();
        echoService.start();

        Socket socket = new Socket("localhost", 7);
        try {
            socket.getOutputStream().write("Hello World\n");

            byte[] buffer = new byte[128];
            int numResponseBytes = socket.getInputStream().read(buffer);
            String response = new String(buffer, 0, numResponseBytes);

            assertEquals("Hello World\n", response);
        } finally {
            socket.close();
        }
    }
}

This test is too broad to make it green in one go. So I need another test. I will start with a server that makes sure all requests are served.

EchoServerTest.java

import org.junit.*;
import static org.mockito.Mockito.*;
import java.net.*;

public class EchoServerTest {
    @Test
    public void delegatesServingTheClientToEchoResponder() {
        EchoResponderFactory echoResponderFactory = mock(EchoResponderFactory.class);
        EchoResponder echoResponder = mock(EchoResponder.class);
        when(echoResponderFactory.newEchoResponder()).thenReturn(echoResponder);

        EchoServer echoServer = new EchoServer(echoResponderFactory);
        echoServer.serveNext(mock(Socket.class));

        verify(echoResponder).respondTo((Socket) any());
    }

    @Test(timeout = 100)
    public void canRespondToMultipleRequestsSimultaniously() {
        EchoResponder echoResponder = mock(EchoResponder.class);
        EchoResponderFactory echoResponderFactory = mock(EchoResponderFactory.class);
        when(echoResponderFactory.newEchoResponder()).thenReturn(new EchoResponder() {
            @Override public void respondTo(Socket socket) {
                try {
                    Thread.sleep(1000);
                } catch(Exception e) {
                    throw new IllegalStateException("Interrupted while sleeping");
                }
            }
        }).thenReturn(echoResponder);

        echoServer.serveNext(mock(Socket.class));
        echoServer.serveNext(mock(Socket.class));

        verify(echoResponder).respondTo((Socket) any);
    }
}

And here is the corresponding EchoServer:

EchoServer.java

import java.net.*;

public class EchoServer() {
    private EchoResponderFactory echoResponderFactory;

    public EchoServer(EchoResponderFactory echoResponderFactory) {
        this.echoResponderFactory = echoResponderFactory;
    }

    public void serveNext(Socket socket) {
        EchoResponderThread thread = new EchoResponderThread(socket, echoResponderFactory.newEchoResponder());
        thread.start();
    }
}

Next I need a server thread that gets a server socket and makes sure every request is served, and that the server can be run in the background. I need this for the acceptance test.

EchoServerThreadTest.java

import static org.mockito.Mockito.*;
import org.junit.*;

public class EchoServerThreadTest {
    @Test
    public void delegatesServingOfClientToEchoServer() {
        EchoServerThread serverThread = new EchoServerThread();
        serverThread.start();

        ServerSocket serverSocket = mock(ServerSocket.class);
        Socket acceptedSocket = mock(Socket.class);
        when(serverSocket.accept()).thenReturn(acceptedSocket);

        EchoServer echoServer = mock(EchoServer.class);

        verify(echoServer).serveNext(acceptedSocket);
    }
}

Here I made the first mistake: The server thread should have gotten the server socket, the echo server and a factory object as constructor parameters. I simply forgot this when writing the test down.

Now I can implement the server thread:

EchoServerThread.java

import java.net.*;

public void EchoServerThread extends Thread {
    public EchoServerThread(ServerSocket serverSocket, EchoServer echoServer) {
        this.serverSocket = serverSocket;
        this.echoServer = echoServer;
    }

    private final ServerSocket serverSocket;
    private final EchoServer echoServer;

    @Override public void run() {
        while(true) {
            echoServer.serveNext(serverSocket.accept());
        }
    }
}

Now I can implement the responder. This class does the real work of sending back data to the client.

EchoResponderTest.java

import org.junit.*;
import static org.mockito.Mockito.*;
import java.net.*;
import java.io.*;

public class EchoResponderTest {
    @Test
    public void canRespondToASingleCharacter() {
        InputStream inputStream = mock(InputStream.class);
        when(inputStream.read()).thenReturn('a').thenReturn(-1);
        OutputStream outputStream = mock(OutputStream.class);
        Socket socket = mock(Socket.class);
        when(socket.getInputStream()).thenReturn(inputStream);
        when(socket.getOutputStream().thenReturn(outputStream));

        EchoResponder echoResponder = new EchoResponder();
        echoResponder.respondTo(socket);

        verify(outputStream).write('a');
    }

    @Test
    public void canRespondToMultipleCharacters() {
        InputStream inputStream = mock(InputStream.class);
        when(inputStream.read()).thenReturn('a').thenReturn('b').thenReturn(-1);
        OutputStream outputStream = mock(OutputStream.class);
        Socket socket = mock(Socket.class);
        when(socket.getInputStream()).thenReturn(inputStream);
        when(socket.getOutputStream()).thenReturn(outputStream);

        EchoResponder echoResponder = new EchoResponder();
        echoResponder.respondTo(socket);

        verify(outputStream).write('a');
        verify(outputStream).write('b');
    }
}

The rest of the classes are pretty simple:

EchoResponder.java

import java.net.*;
import java.io.*;

public class EchoResponder {
    public void respondTo(Socket socket) {
        try {
            int readByte;
            while((readByte = socket.getInputStream().read()) >= 0) {
                socket.getOutputStream().write(readByte);
            }
        } catch(Exception e) {
            throw new IllegalStateException("Interrupted while responding", e);
        }
    }
}

EchoResponderFactory.java

public class EchoResponderFactory {
    public void newEchoResponder() {
        return new EchoResponder();
    }
}

EchoService.java

import java.net.*;

public class EchoService {
    public static void main(String[] args) {
        EchoService echoService = new EchoService();
        echoService.start();

        Object lock = new Object();
        synchronized(lock) {
            try {
                lock.wait();
            } catch(Exception e) {
                throw new IllegalStateException("Interrupted while waiting", e);
            }
        }
    }

    public void start() {
        EchoServerThread serverThread = new EchoServerThread(new ServerSocket(7), new EchoResponderFactory());
        serverThread.start();
    }
}

Come Back on Friday

Here are the steps I had to take to make the system compile and run:

  • It's public class EchoServerThread, not "void"
  • Another syntax error in the class definition of EchoServer
  • EchoReponderFactory: The factory method can not return null
  • EchoService: The constructor of the server thread needs an EchoServer, which in turn needs the factory
  • There's an unhandled IOException in EchoServerThread ("accept()")
  • There's another unhandled IOExcpetion in EchoService ("new ServerSocket")
  • "import java.net.*" missing in the acceptance test
  • The acceptance test has to write a byte array, not a string
  • Static import for asserts missing in the acceptance test
  • And a lot more imports missing... And some more minor syntax errors...
  • EchoServerThreadTest: Must pass the mocked socket and EchoServer into the constructor of EchoServerThread
  • EchoServerTest: The second test has to create an EchoServer
  • EchoResponderTest: have to cast arguments to "thenReturn" to int
  • Unhandeld exceptions in a couple of tests (Just added "throws Exception")
  • 3 Tests failed! The failures were basically timing issues in the tests that deal with threads. Quickly fixed them by adding sleeps (Dirty workaround, I know).

After I fixed all the compiler problems and the timing issues in the tests, I tried to run the service and telnet to it. It worked!

Did I Learn Something?

  • Must of the issues I had were simple syntax errors
  • A huge mistake I made was the constructor call of EchoServerThread in the test and the main method
  • Another big mistake was that I forgot the IOExceptions in several places
  • Getting all the details right is hard. I mean, hard!
  • Refactoring in your head is even harder. I have extracted some classes, but you can find a lot of duplicate. Especially in the tests.
  • I thought this exercise would be hard. It was even harder
  • Even though I made a lot of mistakes, I did better than I expected.

What do you think of this exercise? Are you thinking about trying it? Contact me and tell me about your experiences. If you blog about this exercise, I will gladly link your blog post here and re-tweet it. All my contact details are at the bottom of the page.

Posting Type: 

CSS Vertical Align: Divs

If you have done anything with CSS yet, you probably know the following problem: You want to vertically align some parts of your page, but you do not know the total height of the group. That is, you do not know the height of the largest element. This article shows you how to solve this problem. The approach shown here works in Chrome, Firefox and IE9, I have not tested other browsers.

First, change the box sizing to border-box. Because for me, CSS makes much more sense when box-sizing is border-box.

* { 
    -moz-box-sizing: border-box;
    -webkit-box-sizing: border-box;
    box-sizing: border-box;
}

What doesn't work

Ok, say you've tried the following:

<div style="width: 450;">
	<div style="height: 20px; width: 50px; float: left;"></div>
	<div style="height: 40px; width: 150px; float: left;"></div>
	<div style="height: 100px; width: 80px; float: left;"></div>
	<div style="height: 60px; width: 120px; float: left;"></div>
</div>

Then you've tried to add vertical-align: middle; in various places (at least that's what I've tried for some time). It does not work:

Usign tables... without using tables

What does work: Use tables. But you don't want to use tables in your HTML code. Luckily, you can tell a div to behave like a table:

<div style="display: table-row; width: 450;">
	<div style="display: table-cell;">
		<div style="height: 20px; width: 50px;"></div>
	</div>
	<div style="display: table-cell;">
		<div style="height: 40px; width: 150px;"></div>
	</div>
	<div style="display: table-cell;">
		<div style="height: 100px; width: 80px;"></div>
	</div>
	<div style="display: table-cell;">
		<div style="height: 60px; width: 120px;"></div>
	</div>
</div>

Now, this works. At least it "kinda" works: The divs are aligned at the bottom.

So, what happened here? I have told the outer div to behave like a table row. Then I have wrapped the inner divs (the ones who paint the border and the background) in another div. This wrapping div behaves like a table cell, so it aligns its contents at the bottom.

This new div now can align its contents vertically: I can use vertical-align just like I can use it on a table cell:

<div style="display: table-row; width: 450;">
	<div style="display: table-cell; vertical-align: middle;">
		<div style="height: 20px;width: 50px;"></div>
	</div>
	<div style="display: table-cell; vertical-align: middle;">
		<div style="height: 40px;width: 150px;"></div>
	</div>
	<div style="display: table-cell; vertical-align: middle;">
		<div style="height: 100px; width: 80px;"></div>
	</div>
	<div style="display: table-cell; vertical-align: middle;">
		<div style="height: 60px; width: 120px;"></div>
	</div>
</div>

You can even use different alignments in every cell:

<div style="display: table-row; width: 450;">
	<div style="display: table-cell; vertical-align: middle;">
		<div style="height: 20px; width: 50px;"></div>
	</div>
	<div style="display: table-cell; vertical-align: top;">
		<div style="height: 40px; width: 150px;"></div>
	</div>
	<div style="display: table-cell; vertical-align: middle;">
		<div style="height: 100px; width: 80px;"></div>
	</div>
	<div style="display: table-cell; vertical-align: bottom;">
		<div style="height: 60px; width: 120px;"></div>
	</div>
</div>

This is rendered as:

So, how is this better?

I use divs, but I tell them to behave like tables. You might ask: Why is this better then using tables in the first place?

The approach outlined here clearly separates design and structure. All the design is done in CSS, the structure is provided by simple HTML divs. You can - and should move the CSS to a different file. Then you'll have pure CSS in one file, and pure HTML in another.

Tables, OTOH, provide a structure for tabular data. Using them as a design element is an abuse of tables. The design and the structure would not be clearly separated anymore. But you already knew that ;)

You might also be interested in...

Posting Type: 

Smaller Steps

I always thought that I was doing my unit testing, programming and refactoring in small steps. But during the last year or so I learned that I was wrong. Dead wrong.

I have been interested in code quality and craftsmanship for some time now. The books I read and the people I talked to gave me the first hints that I should do my work in smaller steps. Then, at SoCraTes Conference 2012, I learned about the Taking Baby Steps exercise by Adrian Bolboaca. This showed me how small your steps could really be.

I still don't use the "Baby Steps" technique in my projects - I only use it as an exercise. But since I learned about it, I am trying to take smaller and smaller steps in my projects too.

An example would be handy right now

Today I was refactoring some code I wrote earlier. I had a class that did two things, and I wanted to pull some methods out into a new class. Then both classes would only do one thing and better adhere to the Single Responsibility Principle.

I have simplified the example a little bit. Anyway, the code was something like this:

public class FooService {
    public List listAll() {
        ...
    }
    public List listBy(Name name) {
        ...
    }

    //Other methods that have a different responsibility
}

Step 1: Create a new class FooLists and make it accessible through FooService

public class FooService {
    public FooLists lists() {
        return fooLists;
    }

    public List listAll() {
        ...
    }
    public List listBy(Name name) {
        ...
    }

    //Other methods that have a different responsibility
}

All tests are still green. Of course they are: I only added a new method.
Commit.

Step 2: Copy the two list... methods to FooLists.
All tests are still green. Of course, I only added new methods in a class that is not used yet.
Commit.

Step 3: Change listAll so it delegates to FooLists:

public class FooService {
    public FooLists lists() {
        return fooLists;
    }

    public List listAll() {
        return fooLists.listAll();
    }
    public List listBy(Name name) {
        ...
    }

    //Other methods that have a different responsibility
}

This was the first "real" change. All tests are still green. Phew.
Commit.

Step 4: Find all references of listAll(). Change the first to lists().listAll()
Run all the tests. They are still green.
Commit.

Step 5: Change all other references of listAll()
Run all the tests. They are still green.
Commit.

Step 6: Remove the method listAll() from FooService
Run all the tests. They are still green.
Commit.

Steps 7-n: Do the same for listBy(name). Run the tests after each step.

This was easy

This was really easy. I never had the feeling that I am on thin ice. I always could run "hg revert && hg purge" and only lose the work of a couple of minutes.

I am not yet sure if the end result is what I want. It is definitely better than before. But now there is this train wreck code like "fooService.lists().listAll()" in other parts of the system.

I think I know what I'll do next ;)

You might also be interested in...

Posting Type: 

Softwarequalität für Entwickler

Posting Type: 

Software Quality

"Software Quality for Developers" is a 6-part online course that teaches software developers how to improve the quality of their applications. It teaches topics like developer testing, TDD, mock objects, object oriented design and others. Unfortunately the English version of this course is not available yet - but I am working on it. If you speak German check out the German version here: "Softwarequalität für Entwickler".

Posting Type: 

Pages

My name is David Tanzer and I have been working as an independent software consultant since 2006. I help my clients to develop software right and to develop the right software by providing training, coaching and consultanting for teams and individuals.

Learn more...

Subscribe to RSS - David's Blog