Cursory Screening?

Hiring Criteria and Time for Evaluation in Japanese Firms

Wataru Yoshida and Kohei Toyonaga

National Institute of Population and Social Security Research, Kindai University

2025-07-11

Introduction

College graduated matters

reference

• Applicant’s college selectivity greatly influences hiring decisions (Deming et al. 2016; Gaddis 2015; Rivera 2015; Rivera and Tilcsik 2016)
  • For signals of competence (Spence 1973)
    • or signals of training costs (Thurow 1976)
  • Social ties with employers (Fernandez, Castilla, and Moore 2000; Fernandez and Galperin 2014)
    
  • Homosocial reproduction (Kanter 1977)
  • Cultural fit (Rivera 2015)

And so do test scores for skills

reference

• Standardized test scores are also frequently used signals
  • captures the within-university variations in productivity: Direct signals rather than probabilistic signals (Fossati, Wilson, and Bonoli 2020)
  • standardized criteria for all candidates (unlike GPA)
• The most widely used is the SPI (Synthetic Personality Inventory) test scores
  • provided by a private company (Recruit Co.)
  • cognitive skills and social skills (Protsch and Solga 2015)

RQ1: The effects of university rank and SPI scores on hiring decisions?

No time for consideration!

Hiring staff often have to deal with large number of resumes.

Employers are often faced with dozens, or even hundreds, of applications at a time. “I’d say I look through anywhere from five hundred to one thousand resumes a day,” Carol, a human resources manager in commercial real estate, reports. “Don’t spend more than like three to five seconds on a resume. We kind of just very, very quickly look through them, just glance at them.” (Pedulla 2020, 38)

Then what happens?

Reliance on easy signals

• When time is limited for evaluation, evaluators often rely on stereotypes and other cognitive heuristics to make their assessments. (Kahneman 2011; Rivera 2020)

Reliance on easy signals

• When time is limited for evaluation, evaluators often rely on stereotypes and other cognitive heuristics to make their assessments. (Kahneman 2011; Rivera 2020)

Illustrations

• Non-white candidates face a more severe penalty when applying for internships in software engineering than for full-time hiring. (Campero 2023)
  • For internship hiring, employers are unwilling to invest much cognitive effort in screening, so they tend to rely on group-based stereotypes: Cursory Screening
• Transferring shortlisting tasks from line managers to HR departments ameliorates the gender disparity in hiring. (Sarabi and Lehmann 2024)
  • For line managers, shortlisting applicants is not their primary concern (unlike HR staff), so they invest less cognitive effort and tend to make gender-based judgments.

University rank as status signals

• In Japanese society, the university graduated carries profound social meanings: we refer to these as status signals. (easy to evaluate)
  • Status characteristics linked to the assessment of competence (Ridgeway 2001; Correll and Benard 2006; Correll and Ridgeway 2006)
  • Interpreted under a single ranking system based on admission difficulty (Dore 1976; Kariya 2009; Takeuchi 1995)
  • Prevelent signals not only in hiring but also in many other aspects of social life.

SPI scores as technical signals

• In contrast, we refer to SPI scores as technical signals (takes some to evaluate)
  • No specific social meanings
  • Rarely mentioned beyond the hiring process
  • Takes expertise to understand the average score level and which skills to focus on

RQ2: The relationships b/w evaluation time and effects of university rank and SPI scores

Prediction:

1. Evaluation time ￬ ⇒ the effects of university rank ￪
2. Evaluation time ￪ ⇒ the effects of SPI scores ￪

METHODS

Outline of the experiment

• Survey experiment
  • to accurately estimate the causal effects
  • Possible to measure the time for evaluation (unlike audit studies)

• To 2,095 HR staff working for Japanese companies in Dec. 2023
  • contributing external validity
  • Each participant evaluates 16 fictitious CVs

Outcome

1. Shortlist: What is the likelihood that this student will advance to the next hiring process, such as an interview?
2. Hire: What is the likelihood that your company will ultimately offer this student a job?

• 11-point scale (0-10)

Information on CVs

Factors shown on each CV

Gender, age, college attended, GPA, SPI scores, extracurricular activities, hobbies, and special notes

• School rank
  1. Randomly displaying a combination of universities, faculties, and departments
  2. Assigning a university ranking score (hensachi) published by a private company
  3. Adjusting it to a 0-10 scale
• SPI scores
  • 1 item for basic cognitive skills (1-7 scale)
  • 5 items for social skills (1-5 scale) based on the Big Five
    • extraversion, agreeableness, neuroticism, conscientiousness, openness

Evaluation time

• From starting to show a single CV
• To finishing evaluating for two aspects (shortlisting and hiring)
  • Excluding the top/bottom 1%

RESULTS

Results for RQ1

You can access the full results from here .

• Both university rank and SPI scores matter
  • not only cognitive but also social skills (Protsch and Solga 2015)

Results for RQ2

You can access the full results from here .

• No differences for university rank
• Positive interaction effects for SPI scores and evaluation time
  • cognitive skill (basic) and some of social skills (agreeableness, conscientiousness, openness)

Relative effects of school rank

Compared to an average candidate (university rank = 5, agreeableness = 3), the shorlisting evaluation …

Candidate A (university rank = 7, agreeableness = 3)

Candidate B (university rank = 5, agreeableness = 5)

Relative effects of school rank

Compared to an average candidate (university rank = 5, agreeableness = 3), the shorlisting evaluation …

Candidate A (university rank = 7, agreeableness = 3)

• In 66.7 seconds (95%ile)
  • +0.237 pts higher

Candidate B (university rank = 5, agreeableness = 5)

• In 66.7 seconds (95%ile)
  • +0.303 pts higher

Relative effects of school rank

Compared to an average candidate (university rank = 5, agreeableness = 3), the shorlisting evaluation …

Candidate A (university rank = 7, agreeableness = 3)

• In 66.7 seconds (95%ile)
  • +0.237 pts higher
• In 7 seconds (5%ile)
  • +0.223 pts higher

Candidate B (university rank = 5, agreeableness = 5)

• In 66.7 seconds (95%ile)
  • +0.303 pts higher
• In 7 seconds (5%ile)
  • +0.178 pts higher

Relative effects of school rank

Compared to an average candidate (university rank = 5, agreeableness = 3), the shorlisting evaluation …

Candidate A (university rank = 7, agreeableness = 3)

• In 66.7 seconds (95%ile)
  • +0.237 pts higher
• In 7 seconds (5%ile)
  • +0.223 pts higher

Candidate B (university rank = 5, agreeableness = 5)

• In 66.7 seconds (95%ile)
  • +0.303 pts higher
• In 7 seconds (5%ile)
  • +0.178 pts higher

• Though the absolute effects of school rank do not depend on evaluation time, the relative effects to SPI scores vary.

CONCLUSION

Summary

• Both college attended and standardized test scores matter
  • The impact of both direct/probabilistic signals (Fossati, Wilson, and Bonoli 2020)
  • The impact of both cognitive/noncognitive skills (Protsch and Solga 2015)

• Test scores have a larger impact when evaluators take more time
• Though the absolute effect of university rank do not change, its relative impact gets larger when evaluators take less time
  • Another example of cursory screening (Campero 2023; Sarabi and Lehmann 2024)

Acknowledgements

• This work was supported by JSPS KAKENHI Grant Numbers JP22K13656, JP25K16955, and JP24K16528.

SAQ (Self Asked Questions)

Satisficers

Isn’t it just that people who respond quickly are satisficers?

• We identified satisficers in another question and showed the effect after excluding them.

Those who take very short/long time

Could the results be influenced by a few outliers who took a very short or long time to answer? Given the right-skewed distribution of evaluation time.

• We excluded the top/bottom 5% instead of 1% ⇒ Basically the same result!

The effect of order

Could the order of the items, rather than the items themselves, be related to the interaction effects with evaluation time?

• We cannot rule out this possibility.
  • We did not manipulate the order in which factors were displayed.

• But we believe this is consistent with the actual practice.
  • Employers first look at basic attributes, including the college attended
  • Then review detailed information, including test scores (just our speculation though)
• Assuring the external validity of our result

What social skills mean

Could you elaborate more on the 5 items for social skills?

1. extraversion: leadership, building relationships with many people
2. agreeableness: teamwork, caring about others
3. neuroticism (reversed): dealing well with unexpected change, not worrying much
4. conscientiousness: performing tasks steadily and independently
5. openness: open to new ideas and challenges

Is it a causal effect?

Since the length of evaluation time is not determined exogenously, can you really say that what you found here is the causal effect of evaluation time?

• No, we cannot say it’s causal. (we’re not intend to)
  • Introducing time pressure is an experimental design we would like to try.
• What we found here is a result of …
  • not a joint intervention (the effects of school/scores and evaluation time)
  • but an effect modification (the effects of school/scores by evaluation time)

Determinants of evaluation time

What determines the length of evaluation time? Does any evaluator’s characteristic affect it?

• ICC is 0.41
  • partially depend on evaluator’s characteristics

• Men, young people, those working for large companies tend to take shorter time to evaluate.

Really an effect modification by evaluation time?

Is it an effect modification by evaluation time, or by evaluator’s attributes that affect evaluation time?

• We added the interaction b/w university rank/test scores and evaluator’s gender, age and firm size ⇒ Basically the same result!

APPENDICES

Results for RQ1 (all variables)

Results for RQ2 (all variables)

Literature

Campero, Santiago. 2023. “Racial Disparities in the Screening of Candidates for Software Engineering Internships.” Social Science Research 109 (January): 102773. https://doi.org/10.1016/j.ssresearch.2022.102773.

Correll, Shelley J., and Stephen Benard. 2006. “Biased Estimators? Comparing Status and Statistical Theories of Gender Discrimination.” In Advances in Group Processes, edited by Shane R. Thye and Edward J. Lawler, 23:89–116. Advances in Group Processes. Emerald Group Publishing Limited. https://doi.org/10.1016/S0882-6145(06)23004-2.

Correll, Shelley J., and Cecilia L. Ridgeway. 2006. “Expectation States Theory.” In Handbook of Social Psychology, edited by John Delamater, 29–51. Boston, MA: Springer US. https://doi.org/10.1007/0-387-36921-X_2.

Deming, David J., Noam Yuchtman, Amira Abulafi, Claudia Goldin, and Lawrence F. Katz. 2016. “The Value of Postsecondary Credentials in the Labor Market: An Experimental Study.” American Economic Review 106 (3): 778–806. https://doi.org/10.1257/aer.20141757.

Dore, Ronald Philip. 1976. The Diploma Disease: Education, Qualification, and Development. Berkeley: Univ of California Pr.

Fernandez, Roberto M., Emilio J. Castilla, and Paul Moore. 2000. “Social Capital at Work: Networks and Employment at a Phone Center.” American Journal of Sociology 105 (5): 1288–1356. https://doi.org/10.1086/210432.

Fernandez, Roberto M., and Roman V. Galperin. 2014. “The Causal Status of Social Capital in Labor Markets.” In Contemporary Perspectives on Organizational Social Networks, 40:445–62. Emerald Group Publishing Limited. https://doi.org/10.1108/S0733-558X(2014)0000040022.

Fossati, Flavia, Anna Wilson, and Giuliano Bonoli. 2020. “What Signals Do Employers Use When Hiring? Evidence from a Survey Experiment in the Apprenticeship Market.” European Sociological Review 36 (5): 760–79. https://doi.org/10.1093/esr/jcaa020.

Gaddis, S. Michael. 2015. “Discrimination in the Credential Society: An Audit Study of Race and College Selectivity in the Labor Market.” Social Forces 93 (4): 1451–79. https://doi.org/10.1093/sf/sou111.

Kahneman, Daniel. 2011. Thinking, Fast and Slow. Toronto: Doubleday Canada.

Kanter, Rosabeth Moss. 1977. Men and Women of the Corporation. New York: Basic Books.

Kariya, Takehiko. 2009. “From Credential Society to ‘Learning Capital’ Society: A Rearticulation of Class Formation in Japanese Education and Society.” In Social Class in Contemporary Japan, 87–113. Routledge.

Pedulla, David S. 2020. Making the Cut: Hiring Decisions, Bias, and the Consequences of Nonstandard, Mismatched, and Precarious Employment. Princeton University Press.

Protsch, Paula, and Heike Solga. 2015. “How Employers Use Signals of Cognitive and Noncognitive Skills at Labour Market Entry: Insights from Field Experiments.” European Sociological Review 31 (5): 521–32. https://doi.org/10.1093/esr/jcv056.

Ridgeway, Cecilia L. 2001. “Gender, Status, and Leadership.” Journal of Social Issues 57 (4): 637–55. https://doi.org/10.1111/0022-4537.00233.

Rivera, Lauren A. 2015. Pedigree: How Elite Students Get Elite Jobs. Princeton: Princeton University Press.

———. 2020. “Employer Decision Making.” Annual Review of Sociology 46 (1): 215–32. https://doi.org/10.1146/annurev-soc-121919-054633.

Rivera, Lauren A., and András Tilcsik. 2016. “Class Advantage, Commitment Penalty: The Gendered Effect of Social Class Signals in an Elite Labor Market.” American Sociological Review 81 (6): 1097–1131. https://doi.org/10.1177/0003122416668154.

Sarabi, Almasa, and Nico Lehmann. 2024. “Who Shortlists? Evidence on Gender Disparities in Hiring Outcomes.” Administrative Science Quarterly, October, 00018392241283946. https://doi.org/10.1177/00018392241283946.

Spence, Michael. 1973. “Job Market Signaling.” The Quarterly Journal of Economics 87 (3): 355. https://doi.org/10.2307/1882010.

Takeuchi, Yo. 1995. Meritocracy in Japan [Nippon No Meritocracy]. University of Tokyo Press.

Thurow, Lester C. 1976. Generating inequality. Macmillan.