Earlier this week, a headhunter approached me for a Java back-end developer position. She tells me as you might expect that she has a vague understanding of Software Engineering and that they have two tests to give me before a technical interview. These were a psychological test and a Java Codility test that consisted of three tasks. They gave me three hours to finish the exam. The tasks are usually variations of common programming problems used in academia and programming competitions.
The first and second tasks were very similar just varying in complexity. Codility provides a good web interface with great feedback on what is asked of you. This is definitely a step up from coding on a whiteboard interview. You get two tabs where you can quickly generate test data that is passed into a function in the coding tab. Boilerplate is mostly filled out which is great. Most IDEs do this for you nowadays anyways.
The third task is about finding a bug in the code. The way they test this is checking closely on your reading skills. This is key in modifying a couple of lines of code. The interface did not let me add or remove lines. The solution I came up with require me to add a line so I appended it to closing brace. The interface has no way to allow for debugging so you might need to have your own handy.
Should you filter out developers that do average in these tests?
Probably not. A developer with a decade out of college has already forgotten this kind of coding. Many have never used an array in production. This is part of the reason I got 0% in performance for the second task. I added the array to a list and used methods on the list to solve the task. I did not realized I was being rated on performance and that I had to produce a log n algorithm until I read in the third task that this task will not be rated on performance. At that point, I had no chance to go back. As you can see from my result, performance is 62% of the grade.
CONCLUSION
Codility is not bad but it should not be your only way to measure a developer skill. If a developer is doing data analysis or someone straight out of college, it is probably a great tool since it is more closely related to their experience. However, I was given the impression that if I do bad on these I was not going to a technical interview. I will probably have somebody write a CRUD site or create a cascading drop down, but these things require several technologies which is probably harder to develop a tool to test for. In the mean time, there is not really a good way to measure a developer talent or algorithm to rank them. Use Codility as part of your score, and not the comprehensive examination.