Weblog

Some Thoughts on Testing Developers

For reasons I can't quite fathom, I've been thinking a lot about testing developers recently. That's testing developers as part of the hiring process, as opposed to developer testing (which I do bang on about rather a lot, to be fair).

I say I can't fathom the reasons, because we're not actively recruiting right now, nor am I looking to be recruited (though if you have your air conditioning switched on you may be in luck).

So anyway, it's fair to say that before you hire a developer, you want to find out if they're any good at developing, right? And therein lies the problem: how on earth do you measure the candidate's skill level?

I've seen, and used, a few approaches myself, so I'll go over a few of them and see what drops out the other end.

Approaches I've Used

At PropertyMall we gave candidates a test with a few questions that they could do in their own time, with all the resources - books, internet, cups of tea - that they would have in a real developer role.

The test started off with some really simple stuff about printing out some numbers (so you knew that if they got those wrong you could stop reading). It then went through a few relatively straightforward OO PHP and design pattern examples, and ended with a very open question which gave the candidate a chance to just write some code.

One of the questions I liked most gave the candidate a snippet of code and asked them which design pattern it exemplified. The question was multiple choice, the motivation being that we didn't necessarily need a candidate who could recognise a composite (or whatever it was) straight off, but if they could Google the four terms, most likely find a Java or C++ or Smalltalk example, and relate it to the PHP code in front of them, they probably had the makings of a decent developer.

We hired some bloody good people at PropertyMall, so we were presumably doing something right.

I think the key thing is always to get the candidate writing code. At PlayPhone we just cut to the chase and give a fifty-or-so word written description of some behaviour, and send them home to write the code. If it doesn't suck, we invite them back to discuss it - and that's probably the most important part of the test: having the developer explain their approaches, thought processes and motivations.

Approaches I've Come Up Against

I've changed jobs a handful of times myself, so I've come across a few different approaches to testing developers.

Like absolutely every other PHP developer in London I've been put through Brainbench by Allegis, who recruit for IPC. Brainbench is taken online (so with access to the web a requirement, rather than an option!) and is time-limited. I think it was about 45 minutes, and the questions come thick and fast. At the end you're left with a cold, hard numerical score, which is presumably for the benefit of managers and other folks who can't grasp anything complex or organic.

The vendors claim that the test is smart, so the better you're doing, the harder the questions get. I did the PHP and the Perl ones, and I seem to remember the results were quite complimentary about my PHP. However, the system is clearly flawed, as it put me in the top few percent of the nation's Perl developers despite me blindly guessing my way through most of it, grimacing occasionally as I struggled to recall the few dozen lines of Perl I wrote as a student.

Another approach I've come across was that the company would email over a document with a bunch of questions or exercises, you'd have a crack at it, and 45 minutes later they would ring you and discuss it all. That's quite a nice combination of time-limited and open-book, it incorporates the all-important discussion stage, and doesn't waste too much of anyone's time.

Well, I say that, but it turned out that the company involved were such amateurs that they twice failed to keep the appointment, before I got fed up and told them to quit the hell wasting my time. But I liked the idea.

Closed-book can be interesting too. Before I started my current job I interviewed with a promising startup who were looking for a lead developer to head up the technical side of the company. That's quite a challenge and a lot of responsibility, so they were keen to make sure that they got the strongest developer they could find.

The result was a very challenging but rather satisfying time-limited, closed-book test covering everything from hardware performance to regular expressions, dependency injection and even DSLs - not your typical PHP web scripting stuff, for sure.

What surprised me was not my score, but the fact that it was expressed as a percentage. I think I managed something in the low 80s, but the implication that they thought that any of that stuff had a right or wrong answer really set the alarm bells ringing.

It got worse though, and the post-mortem was painful as hell. Now, since the company didn't yet have a technical team (that would have been my job to remedy), they had hired a local "guru" (surely I'm not the only one who raises an eyebrow when that word crops up?) in a kind of consultancy role, to set the test.

In the post-test discussion and interview, it emerged that he'd marked me down for doing a command line svn merge using a syntax with which he was unfamiliar. He was also completely unaware of svnserve, and rather assumed I was making it up; another black mark there. I was given a dressing down for suggesting that PHP makes for a perfectly good templating language (that's only what it was invented for, mate) and again for once having written my own framework. Conversation later turned to the framework he was writing.

The whole thing left a bitter aftertaste, and after a good night's sleep I'd decided I really, really didn't want the job. Conveniently enough, I didn't get it.

But I digress. I don't want to turn this into a personal rant, because - believe it or not - I do actually have a point here, which I can best sum up as:

Who tests the testers?

Really, what reason does the candidate have to believe that the tester is any better than them, or even that they have a clue what they're doing? As you climb the experience ladder over the years, that question becomes more and more pertinent with every rung.

I think this is why I'm uncomfortable with handing out tests that are supposed to have a right and a wrong answer. On the other hand, I don't think that personal opinion (as in the templating example above) has a role to play in testing developers either.

Conclusions, If Any

So you can see what a minefield this can be, and why after eight years of interviewing and being interviewed, this is a nut I'm still to crack.

I guess I've picked up a few crumbs of wisdom over the years though. I definitely feel that open-book tests are the way to go, since they more closely simulate the environment in which developers actually work. I think all tests should be done with an internet connection, at the very least.

I also think it's utterly vital that a test doesn't seek to trick a candidate, or catch them out. You want to know what they can do for you, after all, so give them space to show it.

I think you also want to find out if the candidate is bright, and so you'll want to throw something in there that tests their brain, rather than their experience of a specific programming language. Not for nothing is Joel's book titled "Smart and Gets Things Done".

So what about you? How does your company test developers, and what experiences have you had? Finally, what role does professional certification, such as Zend Certification, play in all of this?

Response to "10 Things a Developer Should Never Ignore"

Earlier this week, I stumbled across Bill Stronge's recent 10 Things a Developer Should Never Ignore over on TechRepublic. It's recommended reading, as it's an interesting piece, filled with useful advice for developers, especially those just getting started in their programming career.

Still, a couple of the points jarred with me a little, and there were a couple which I felt could have been taken further. So here's my response to Bill's 10 Things.

#1: Clarifying User Requirements

I agree with this point in principle, but it's important not to get bogged down in the requirements gathering phase. Try to understand your customer, but don't expect them always to know what they want before they actually see a product coming together.

Furthermore, expect requirements to change over time. This is a constant source of frustration for inexperienced developers. I know because I've been there myself, but over time you will come to expect it and embrace change.

I think this is where agile processes win hands down. Agile urges you to keep it simple, roll out small, incremental changes to a system, and adapt to changing requirements over time. The result is almost always a more valuable product, happier customers, and more satisfied developers.

#2: Collaborating

Agreed. We've all worked with the "hero" programmer, who churns out thousands of lines of code, and jealously guards it from the eyes and - heaven forbid - input of his or her colleagues.

Unfortunately, the only possible outcome of that mindset is libraries that no one else wants to use, and code that no one else wants to maintain.

You don't have to go as far as pair programming (though there's a lot to recommend about it), but just don't let your pride get in the way of talking your ideas through with other developers, asking questions and listening to advice.

#3: Version Control

Agreed. Get your code, configuration files and even documentation into version control and feel the weight lift from your shoulders.

Furthermore, learn how to use version control well. Learn how to use tags and branches properly, and read up on more advanced features, such as svn externals. On this subject, I strongly recommend the Pragmatic Version Control books.

Finally, if you're only familiar with a graphical VC client, such as Tortoise, learn how to do it all from the command line. You'll find it quicker, simpler, and a great deal more powerful.

#4: Basic System Testing

I would take this point a lot further and recommend unit testing and Test-Driven Development. In short, unit testing is writing code to test your code, and Test-Driven Development (TDD) is simply writing the tests before the code.

Bill states that most developers hate testing, but I believe that they only think they do. Programmers love programming, and I like to think of unit testing/TDD as a programming technique rather than a testing technique. In fact, designing and coding a thorough and yet flexible test suite can often be as satisfying a challenge as the project you're actually delivering.

The investment of a little time up front to write some small tests typically pays back a hundredfold in terms of productivity. Time spent debugging is slashed, and you have the pride of delivering code in which you have complete confidence. Furthermore, code built using TDD invariably tends to be cleaner, terser and more flexible than testless code.

Once you become what's known as "test-infected" you're very unlikely ever to voluntarily return to the pain of untested and untestable (also known as detestable) code!

#5: Usability

It's hard to argue with this, but it's even harder to define 'usable'.

Programmers are notoriously hopeless at usability, especially the ones who think they're great at it. You only have to look as far as the clunky Ajax-heavy interfaces of trendy Web 2.0 sites such as Spoonfed [1]for evidence of that. Someone clearly had a whale of a time coding that site, but it's painful for the end user.

Being a programmer myself, I don't have a great number of pearls of wisdom to share when it comes to usability. The best advice I can give is to exercise some self-restraint, keep it simple - boring even - and your users will be far less likely to want to strangle you.

#6: System Performance

Nobody likes slow sites and applications. But please, please pay close attention to the three rules of optimization: Don't, Don't yet, and Profile Before Optimizing.

I don't know if I'd go as far as the commentator who is quoted as saying:

More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity

After all, blind stupidity is pretty prevalent. But I certainly have seen some terrible code written, and some deeply regrettable architectural decisions made in the name of optimization. Quite often those "optimizations" tend to lead to far worse performance.

In my experience, two criteria must be in place before you can justify attempting any kind of optimization: i) you must be able to prove that you have a performance problem - anything else is a waste of your time and a waste of your employer's money; and ii) you must be able to measure the problem. If you can't measure the problem, you can't prove that your whizzy optimizations haven't just made things worse.

#7: Comments in Your Code

Time and time again, we're told that comments are a Good Thing, full stop. But the situation is a little less black and white. Comments need to be appropriate.

We've all seen this kind of thing trotted out as an example of poor commenting:

That's plainly stupid, but examples like this tend to miss the real point.

As a rule of thumb, if the code you're writing is complex or obscure enough that you need comments simply to explain what it does, then that piece of code has bigger problems than can be solved just by adding a comment.

In his now classic book, Refactoring, Martin Fowler lists comments as one of his "bad smells in code" for this exact reason. It's much better to strive for shorter, clearer classes and methods. Give them expressive, meaningful names and signatures, and they can become almost self-documenting.

It is however valid to use comments to explain why code does what it does. That line of code that adds 1 to $x may do that to fix a critical bug. In this case a brief comment highlighting the fact, perhaps with a reference to a bug tracker id, will signal to the programmers who have to work with this code in future exactly why that line is there.

#8: Logging

I would urge that Bill's advice to build some helpful logging solutions into the code be followed with great caution. It may help in some specific cases, but spurious logging code can clog up the real application code and make it a great deal more difficult to read.

Furthermore, there's very little that you can do to anger your systems guy more than sending pointless logging code into production, and eating up disk space with log files that nobody ever reads.

#9: Keeping Your Skills Up-to-date

Agreed. Strive for the deepest possible understanding of the tools and technologies you're using, but also read around the subject. Pick up a new language every now and again, read about design patterns and get the hang of some Unix tools, such as sed and awk. Play with benchmarking tools and try to make it to developer conferences and local user groups.

You'll be ten times the programmer you would be otherwise, and you'll thank yourself in that next job interview, because what do you do to keep your skills up-to-date? is a question that's guaranteed to come up.

#10: Taking Pride in Your Work

Agreed. See points 1-9 for more details!

Footnotes

[1] It's only fair to point out that Spoonfed has been redesigned since I linked to it. The new version is greatly improved in some ways, but unfortunately still makes hopelessly gratuitious use of Ajax, which greatly detracts from its usability.