Finally, Some Good News of the Replication Front – Sorta

While the failure to replicate findings from the psychological literature has been a common critique of psychology in the recent press, one area of psychology which does appear to replicate is that of trait-based prediction, a finding that is especially relevant for I/O Psychology. As I/O psychologists, one of our key tools for prediction is trait-based personality tools and cognitive assessments. The findings from studies investigating the relationship of these tools to life outcomes, such as job performance, tend to replicate over and over again. Good news!

However, replicated findings cannot be interpreted without first understanding the power and meaning of those findings. Replication is important as it provides confidence that findings have some robustness. Application of the findings does, however, require some caveats:

  1. The prediction of behaviour is inherently difficult, and the trait measure noted only predict a portion of the variance related to job performance: Correlations of 0.3-0.5 indicate that 9-25% of the variance is accounted for by these measures, leaving plenty of unaccounted variances.
  2. Studies that use a large sample size can lead to misinterpretation when one applied the findings to individuals. The correlation represents an average of the relationship across a range of individuals. If you are hiring a large group of individuals, playing to the averages makes sense. However, often organisations are hiring a single individual, and this is where a prediction of that person’s performance from that individual score becomes much harder.There can be a huge variation in the predicted score of a given individual even when a researcher has obtained an acceptable correlation. To indicate this phenomenon I suggest people look at the graphic in the from the following slide show (slide 16). Remember that a 0.3 correlation is considered high.
  3. If a practitioner is relying on research to inform how they will use a trait-based tool, they will be forced to conclude that a higher is likely to result in better job performance. Ideal profiling makes sense when you are wanting to add nuance to the selection process (such as using the test to probe further on perceived gaps), but ideal profiling is counter to the linear predictions made by research.
  4. In line with point 3, deriving competencies in light of replicated research become difficult to justify based on scientific grounds.
  5. There may be negative relationships between valid predictors, and the practitioner may need to account for combinations of traits. An example of this is the relationship between consciousness and cognitive ability when assessed in candidate samples.
  6. The findings that replicate indicate that people that work hard, are detailed oriented, reliable, emotionally stable and are intelligent are likely to perform better at work. Findings such as these qualify for what the famous Michael Scriven would attribute to Grandma’s law (Grandma could have told you), That said, the advantage is that unlike Grandma we can measure these traits!

So in summary, the replication of the predictive power of traits for determining life outcomes is indeed something to celebrate. However, we should temper the celebrations mindful of the limitations of what we are replicating. Psychology is indeed a science, and philosophy and the prediction of human behaviour is, by its very nature a highly complex equation with many unknowns.

References

“Micheal Scriven“ (2019, April 11). Retrieved from https://en.wikipedia.org/wiki/Michael_Scriven

“Most Links Between Personality Traits and Life Outcomes Are Replicable, Study Shows” (2019, April
11). Retrieved from https://www.psychologicalscience.org/news/releases/personality-traits-life-outcomes-replication.html

“Myths and realities of Psychometric Testing” (2019, April, 11). Retrieved from
https://www.slideshare.net/OPRA-Consulting-Group-2015/myths-and-realities-of-psychometric-testing

Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and performance at the beginning of
the new millennium: What do we know and where do we go next?. International Journal of Selection and assessment, 9(1‐2), 9-30

Frank L. Schmidt (2002) The Role of General Cognitive Ability and Job Performance: Why There
Cannot Be a Debate, Human Performance, 15:1-2, 187-210, DOI: 10.1080/08959285.2002.9668091

Paterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2016). An Assessment of the Magnitude of
Effect Sizes: Evidence From 30 Years of Meta-Analysis in Management. Journal of Leadership & Organizational Studies, 23(1), 66–81. https://doi.org/10.1177/1548051815614321

Wood, P., & Englert, P. (2009). Intelligence compensation theory: A critical examination of the
negative relationship between conscientiousness and fluid and crystallised intelligence. The Australasian Journal of Organisational Psychology, 2, 19-29.

2019-04-12T05:41:39+00:00