A 4th hurdle might be noticing that, actually, you only need to check the first birth year through the last birth year. You dont need to check years in which people only died; they certainly wont have the highest population.
Sam Pullara, CEO of Wavefront and formerly Yahoo and Twitter
Ive worked with a lot of great programmers over the years who dont have computer science degrees. Their biggest disadvantage is that they often lack the jargon of computer science.
Given an array of people, with their birth and death years, find the year with the highest population.
For example, consider this question, which has a bunch of solutions, each of which presents a new hurdle:
So far, weve been focusing on the diverse categories of challenges, based on what you need. Now lets get down to the practical ways of designing challenges. We looked at the question banks of various companies to pinpoint the patterns of what makes a code challenge successful. Based on these patterns, heres a checklist of 5 common mistakes to avoid and ensure each challenge will draw the best candidates:
Theres a common misconception that you need to have a computer science (CS) degree in order to be a good programmer. In reality, only 40% of working software engineers in one survey said they had a CS degree.When we wrote about this inTechCrunch, Wavefront CEO Sam Pullara told us he agreed.
For good measure, test out questions on your peers first. Ask at least 2-3 engineers to solve the problem before unleashing onto your candidates. This helps you see what sort of hurdles are in the question and what to expect from candidates.
In an algorithm code challenge, you should have at least one challenging question where only about 20% of your candidates solve it perfectly.A good algorithm question would have 2 (or more) different solutions, one more optimized than the other.This leads to 3 tiers of performances:
Before You Do AnythingDesigning ChallengesChallenge Checklist
This sounds like a no-brainer. But youd be surprised as to how many companies just get their own code challenge wrong. CareerCup CEO Gayle McDowell frequently consults with tech companies of all sizes on their hiring process.
Senior candidates, particularly, are most likely to be unfamiliar with CS textbook words. For instance, avoid dropping terms like state machine or dependency injection. Likewise, in order to avoid ruling out senior engineers who havent taken CS fundamental classes (or dont remember them), avoid questions that involve more obscure algorithm knowledge.
If you use a platform like HackerRank, for instance, you can evaluate based on the time to complete a test case, taking into account both asymptotic complexity and the lower constraints. But it doesnt impose a time condition on all problems. Its up to you as the problem designer to decide if the efficiency is important.
We did an experiment in which we compared two similar companies challengesone with a generic problem statement and another with a more tailored challenge. We saw a nearly 10% higher completion rate with the latter. Even anecdotally, we consistently see this direct correlation time and time again.
This brings us to not only the most important step but also the most overlooked component of hiring senior engineers.
Saved VMware engineers 100+of hours per quarter by only calling candidates who passed the coding challenges
When it comes to algorithm code challenges, McDowell estimates that if 5% of your candidates instantly know the solution to your question, then its likely to be too easy.Easy challenges cluster the mediocre and the great, which of course makes identifying great developers very difficult. Your challenge should be hard enough to separate the average from the top.
correct solution. Most candidates should be able to come up with a brute force algorithm (e.g., for each year that a person is alive, walk through all the people to count the number of people alive in that year), but a few wont. If they dont, give them some help and see if this was just a brief lapse in reasoning or a consistent issue.
Candidates are more drawn to and interested in challenges themed to your mission versus generic, cold challenges.VMware, for instance, created a host of very detailed, complex problem statements that delved deep into virtualization. You can see theLogical Hub Controllerproblem is richly tailored to a typical virtualized datacenter problem they grapple with daily. By replacing resumes with tailored code challenges for two years, VMware:
Tier 1: The best candidates will pass all test cases because they implemented a correct and optimal solution.
If a candidate gets stuck on some aspect of a problem, were they just unlucky? Its important to distinguish between these factors, and the best way to do this is by getting multiple data points. That is, ask a question that involves multiple hurdles. This way, if a candidate gets stuck on one aspect, you can help them through it and they still have other logical leaps to overcome (and thus additional data points for your evaluation).
Tier 3: The candidates you dont want to hire wont develop a correct algorithm at all.
Engineers who can identify the right sub-problems, solve smaller problems and then merge them together are more likely to be successful as leaders in your org. Being able to break things down, spot patterns and attack problems systematically are crucial for any strong engineer. The more tailored and specific the problem statements are to your industry, the higher the chances of high-quality candidates completing code challenges as part of their application.
And so on, until we arrive at an optimal solution. Great candidates might even leap past several hurdles at once, and thats a great sign. A problem like this has so many hurdles that you can more effectively see if theres a pattern in the candidates performance.
And its not a matter of carelessness or minor bugs in typing up the solutions. The challenge designer genuinely believed their algorithm was right until they were proven wrong. Take the time to ensure that 100% of your answers are correct. Otherwise, youll turn away a lot of strong candidates who were actually answering the question right.
When Ive reviewed companies question banks, about 10% of answers are wrong, McDowell says.
Third hurdle:A 3rd hurdle might be noticing that you only need to check the years between the first birth year and the last death year. So a candidate might first get that information, and then proceed with checking that range of years.
Increased the success rate for candidates who come onsite (compared to manual resume screenings).
Tier 2: The okay candidates will pass most of the test cases with an algorithm thats correct but suboptimal. Their codegives the right output, but times out on the large test cases.
This brings us to not only the most important step but also the most overlooked component of hiring senior engineers.