Updates

By Wendy Beach, VP of Talent Science Solutions at Stang Decision Systems

Spoiler alert: here comes some real-life science to distract you from the daily news! In all seriousness…there is beauty, I’ve found, in the science behind what helps our company match people to jobs and jobs to people.

Because it’s good to understand why we do things a bit differently, this blog on choice-supportive bias is our first in a series of discussing the relevant cognitive biases which prevent many people from making good hiring decisions.

Once we make a choice, we have the tendency to support that choice (dig in our heels, close our minds, stick our heads in the sand…insert your favorite saying here) even if it doesn’t make sense in the face of new information. Industrial psychologists call this choice-supportive bias, and it is one of many cognitive biases that affect our decisions.

I would argue that choice-supportive bias has become even more significant as our world has become more competitive and information leading to constant decisions is available at an ever-increasing pace.

What!? Isn’t it a contradiction to say that in an increasingly fast-paced and competitive business world the “status quo” is becoming even more ingrained in our personal and professional lives? I’d argue we are living in a world of “revolving status quo.” In other words, we try even harder to maintain normalcy and consistency in certain parts of our lives because other parts are changing so quickly.

The two main reasons for this are: (1) the increase in disruption via information and technology available at any moment; and (2) choice-supportive bias. The information and technology pushes us to change, and choice-supportive bias pushes us to lock in previous choices. The conflict between the two leads to pockets of rapid change and pockets of locked-in behavior. Oddly, this implies that rapid change in many areas of business run parallel to systems that may be archaic or obsolete.

Anyone in business, from manufacturing to service, has every moment of the day filled with constant choices and decisions. Since our company is focused on talent, I’m going to simplify this conversation (whew!) and emphasize the importance of resisting choice-supportive bias when attracting, retaining and developing talent. Specifically, I’m going to use the resume as a shining example of how cognitive biases can ultimately reduce decision accuracy.

“Eyeball to eyeball and a handshake.” Does anyone else feel a bit of nostalgia for the times when this was the way you decided if a person was a good fit for your team? In the 1940’s, believe it or not, the resume was a new trend, and it typically included weight, age, height, marital status, and religion (not very PC!). It became popular as companies grew and roles became more skilled or specialized. Resumes also helped as workers had more types of work experience and more geographic mobility. For a time, this was an improvement that made a lot of sense.

The reality of today, however, is quite different. In this fast-paced world, most people do not want to spend hours crafting the perfect resume. What if a good candidate simply doesn’t use the right keywords, or has a job title or education that doesn’t describe his or her full skill set? What if some candidates pay graphic artists and editors to make their resumes shine in a way that others never will?

From the perspective of the hiring manager, who wants (or has time) to spend hours sorting through hundreds of resumes–or worse–resumes with cover letters? Hiring managers and human resources staff are often faced with mountains of paper and/or electronic documents in varying formats, fonts, and length. Nonetheless, thanks in part to choice-supportive bias, the resume has become the default tool for evaluating candidates. Resume, cover letter, status quo, check.

When a “new” idea came along in the not-too-distant past to allow technology to help us with the tedious task of going through resumes, we didn’t start from scratch and look at what computers and data could truly do to elevate the process. Instead we, as a society, collectively held on to the “status quo” while adding an electronic layer. Our “choice-supportive bias” resume protocol was still intact, and the document was simply stored on a computer.

Resume parsing, keyword searching, arbitrary cut-offs of grade point averages…those were all born of using a computer to get through big stacks of documents faster. Did that increase our odds of a great fit? Actually, even though electronic applicant systems were perceived to increase efficiency, they quickly become talent pool limiters, and often increase inefficiencies when it comes to finding the most suitable candidates for a job. There are many internet forums that focus on how much candidates hate applicant tracking systems, and internal human resources staff tend to agree.

What if we used computers and data to help us get to know people better as a first step in a hiring process, not simply weed them out? Can computers really help us be MORE human?

You’ve probably guessed by now that using technology to make the hiring process more accurate and more “human” is our focus at Stang Decision Systems. Our clients are bypassing resumes altogether and working with new ways to see their candidates as a complete person. Curious how? We’d be happy to show you.

When the resume step is eliminated, we start to wonder why we held on to it for so long. Could it be that we’re simply human, with some pesky choice-supportive bias? I think so, but I’d love to hear other theories.

As with any limiting quality, we should not expect to be other than human, or beat ourselves up over it. What we can do, as intelligent decision makers, is become more aware of our biases and search for ways to overcome them. The human brain has a wonderful autopilot mode, but when decisions are truly important, we need to take the wheel and steer toward better options. That is truly progress beyond the status quo!

As I’ve talked about in a previous blog post, a primary goal that drives our work at SDS is to achieve a 90%+ selection process accuracy rate. In other words, we want more than 90% of the people who are hired through our tools to be considered “good” hires.

In order for us to determine whether we have met that goal, we need information about how well these people are performing on the job. However, collecting information about people’s on-the-job performance can get tricky. It’s not always easy to determine with confidence who is a “good” hire and who is not. There are many questions we ask ourselves which help us to trust our performance evaluation data:

• What types of information do we use to determine whether or not someone was a “good” hire?
• Whose input do we consider to be the most useful?
• How is this information collected?

The answers to these questions are critical factors which help us when determining if our selection process is resulting in “good” hires. I’m going to focus this blog on explaining some of the most critical factors we consider when collecting performance ratings.

CONTEXT MATTERS

When supervisors are asked to make evaluations on their people, the context in which these ratings are gathered can dramatically influence the true accuracy of the evaluations. For example, if a supervisor is asked to make evaluations that will be used to determine pay and promotion, the supervisor may rate someone differently than if they were evaluating a candidate for purposes of determining future training. In either of these cases, there are motivations that affect how a supervisor evaluates their subordinates that may have little to do with making precise, accurate evaluations. These motivations may include factors such as maintaining the morale of the supervisor’s team, motivating people to improve in the future, or ensuring subordinates receive pay increases.

Because of these motivations, we prefer to not use ratings obtained within an organization’s performance appraisal system. Instead, we prefer to collect our own ratings with a survey that is ONLY used for purposes of evaluating the selection process. We let evaluators know that their ratings of each individual will not be seen within the organization, and will be used for research purposes only…which ultimately leads to continuous improvement in their selection process. By doing this, it allows supervisors to focus on making accurate evaluations of their people without having to worry about the implications of these ratings.

PROCEED WITH CAUTION WHEN USING OBJECTIVE CRITERIA

For many jobs, people are evaluated by objective criteria, at least in part. For example, a production worker in an assembly plant may be evaluated by number of products assembled; a police officer may be evaluated by number of tickets written; or a car salesman may be evaluated by number of cars sold. On the surface, these criteria seem very reasonable. However, as Borman points out, there are several reasons why objective criteria often do not accurately reflect someone’s true performance on the job.

First, objective criteria may only reflect a small part of one’s job (e.g. the number of tickets written by police officer). Second, when using objective criteria, someone’s performance is often dependent on factors that are outside of his/her control (e.g. production worker is dependent on many others when assembling products). Finally, the numbers obtained from objective criteria might be difficult to evaluate. For example, a car salesman at one store might sell the same number of cars as a salesman at another store, but the market for cars at these stores could be very different.

Ultimately, when using objective criteria as a performance measure, you need to diligently research how these measures are obtained in order to ensure that the information truly reflects the person’s performance.

SUPERVISOR’S PERSPECTIVE DOESN’T ALWAYS TELL THE WHOLE STORY

When collecting performance ratings, the supervisor is often considered to be the person who is in the best position to make these evaluations. However, there is often value in collecting ratings from sources in addition to the supervisor. As many jobs are complex in nature and include working with people at all levels of an organization, ratings from sources such as peers, subordinates, customers, etc. might capture information about someone’s performance that isn’t reflected in supervisor-only evaluations. By collecting ratings from multiple perspectives, a more comprehensive portrayal of employee performance can be obtained.

Clearly, there are many issues to consider when collecting performance information. While the purpose of this article was to introduce some of the most common issues that we often wrestle with, we realize that each organization and each job has its own factors that need to be considered. As such, we realize that there is no perfect measure of how someone is performing, which in turn can make it difficult to determine who is a “good” hire. What we do know, through experience, is that following best practices for collecting performance ratings can greatly enhance the accuracy of this information.

The status quo is broken. Specifically, resumes, unstructured interviews, and combining predictive measures using professional judgment doesn’t work well. Over one hundred years of research in the area of industrial and organizational psychology supports this thesis, so I’m going to skip the basics and dive into the good stuff.

The good stuff, in my opinion, is more toward the cutting edge of decision science. We know that human beings, on their own, are imperfect decision makers. We know that well over 100 documented cognitive biases have an effect on our ability to make optimal decisions. We also know that many decision tools have demonstrated the ability to improve the accuracy of hiring decisions.

Job simulations, cognitive assessments, structured interviews, well mapped personality assessments, situational judgment tests, biodata, and physical ability assessments are all potentially valuable for predicting job performance. Moreover, when we combine the results from multiple job-related assessments in a statistically optimal, or even reasonable, fashion, the overall prediction is far more accurate, on average, than predictions made by experts who rely strictly on their “guts” to make these decisions.

To re-phrase the last sentence, algorithms beat experts at predicting future job performance. This general finding has been tested hundreds of ways with different types of predictions and different types of experts. The results are powerful, they are conclusive, and they are massively underutilized in the real world. We aim to help change that by including statistically optimal scoring generated by carefully derived algorithms with all of our assessment results.

So, for example, if you have a candidate who has taken three assessments, we will provide you with four scores:

Works Hard = 9.1
Works Smart = 6.4
Works Safe = 8.5
Baseline = 8.4

The first three scores indicate the individual assessment results and the final score indicates the overall score, appropriately weighted, across the assessments. We call this weighted composite score a Baseline score.

The Baseline score is the single best indication of a candidate’s probability of success on the job and Baseline scores are directly comparable across candidates. So, in a situation where different candidates have different strengths, the Baseline score can be used to quickly and accurately rank order the candidates in terms of their probability of success on the job.

Think about employee selection decisions that you have observed. Most hiring decisions come down to a person or group of people trying to compare candidates in an apples and oranges fashion. Candidate Joe has the most appropriate college degree for the position, candidate Sue has better job experience, candidate Pat had the best energy level in the interview, and candidate Kyle scored highest on the math test.

Who should you pick? How much weight do you put on each of these factors and how do you combine them to look at the “whole person” and compare that person to the needs of the job? Does a college degree really matter for this job? Is job experience at one organization easily transferable to another? Does “energy level” in a 30-minute interview suggest high energy on a day to day basis? Does “high energy” really matter if the person isn’t smart enough to be trainable?

People who have been involved in hiring decisions readily see that the complexity of most decisions quickly goes beyond our abilities. Fortunately, the human brain has wonderful mechanisms for dealing with complexity. One of these mechanisms is the use of simplifying strategies commonly referred by decision making researchers to as “heuristics.”

Our brain knows that, on average, a loud noise is more important than a soft noise. Things that smell good are more likely to be edible than things that smell nasty. A restaurant with many cars in the parking lot is more likely to be a good option versus a restaurant with few cars. In all of these cases the rule of thumb has some merit, but it is also likely to lead us astray at times.

Imagine that you are in charge of picking players for an NBA basketball team. There are lots of players from all over the world to choose from so you decide that you aren’t going to look at anybody who’s height is under 6’. You know that there are some great players who are less than 6’ tall, but you also know that you don’t have time to evaluate every player and by cutting out all people under 6’ your pool of candidates seems much more manageable.

Cutting part of the pool allows you to focus your time on players with the highest probability of success at the expense of a tiny proportion of great players who are under 6’ tall. This same scenario applies for any type of minimum requirement when hiring. Requiring 5 years of work experience, a 3.0 GPA, or a college diploma will simplify your hiring decision . . . at the expense of precision. What if we could simplify our options without losing precision . . . wouldn’t that be a better option?

Source: http://www.nytimes.com/interactive/2013/11/03/sunday-review/so-you-want-to-play-pro-basketball.html?_r=0

As decision makers we face a dilemma. We need lots of data to make an accurate decision and we need to simplify that data to make sense of it. Simplifying is smart, which is why our brain does it automatically using heuristics. Over simplifying, however, is not smart and leads to many preventable mistakes, which is why we need a better way to make important decisions.

Algorithms provide the better way. An algorithm can efficiently combine data from a wide variety of measures in a fashion that minimizes data loss. An algorithm can be used to quickly and accurately rank order 7000 candidates in seconds and the results are far more accurate than if a team of people scour the same information for weeks. Algorithms can also be tracked over time and updated with improved algorithms.

Oddly enough we can model the decision making of experts using policy capturing studies, and the resulting algorithms are more accurate than the people they were modeled after (crazy, but true). In fact, we have modelled the decisions of NFL teams and created algorithms that rank order NFL draft prospects. On average, these algorithms are 36% more accurate than the selections made by actual NFL teams. We’ll get into the details of why this is true in future blog posts, but the simple explanation is that algorithms are extremely consistent whereas humans, even expert humans, are not.

So, when you think about humans being beat by algorithms, how do you react? Anger . . . “they” can’t be better than us! Fear . . . what are people going to do if the algorithms start making all the decisions? Disbelief . . . what does this guy know, he’s just a guy writing a blog. Or amazement . . . imagine the great things people could do if they improved their decision making when making life changing decisions! Many years ago I was a bit irritated that computers were beating Chess champions and, much later, Jeopardy champions. Today I have come full circle and I am simply amazed that people can build machines, computers, and algorithms that make people better at doing things they love to do.

Just as a bulldozer amplifies the power of a person and allows her to move more dirt faster than a hundred people digging with their hands, algorithms allow us to make faster, more accurate decisions in a less biased manner. The initial fear and pushback against using algorithms to improve our decision making is both understandable and irrational.

Let’s get over it and start making better decisions.