From child custody to sentencing to mental capacity, psychological assessments in legal contexts can literally decide matters of life and death.

Now, a new study in the journal Psychological Science in the Public Interest questions the reliability of many such tools currently in use.

In most legal settings, judges decide which expert tests and testimony to admit as evidence, but many jurists lack the means to evaluate the quality of the science behind the evidence.

Strong evaluation tools should undergo rigorous peer review, provide a known error rate and include well-defined cases in which they should or should not be applied. But is that really the case?

"If some psychologists are using strong methods and some are using weak methods, are the courts sensitive to that? Are they screening out the stuff that's weak, and are they letting in the stuff that's strong, or are they letting everything in? Are they screening everything out?" said lead author Tess Neal, an assistant professor of psychology at Arizona State University.

To see how well the judicial system handles psychological assessments, Neal and her colleagues used 22 surveys to identify 364 psychological assessment tools used in legal cases.

They then looked at how often, and how effectively, such tests had been challenged.

Researchers found 90 percent of the tools in question had undergone scientific testing, but only about 67 percent were considered by experts as generally accepted in the field.

Worse, only about 40% had received favorable reviews from experts.

"That does tell us that there's variability in the scientific quality of these tools, and that the courts really should be paying attention to, and perhaps challenging, the tools when appropriate – and definitely screening them out when appropriate," said Neal.

Lawyers challenged psychological tests in only around 5 percent of the cases sampled. Only one-third of the challenges successes.

Previous research has shown courts regularly admit scientific evidence lacking in clear or solid foundations. Such oversights can ultimately undermine the cause of justice and confidence in the legal system, as occurred when experts called into question the validity of some forensic evidence.

Earlier work has also revealed many educational and psychological tests have not undergone rigorous validation. According to the paper, professional reviewers have described the quality of a high percentage of tools in circulation as "unfavorable, mixed or neutral."

In short, psychological tools, tests and instruments differ substantially in validity and applicability. To take an example from the paper, intelligence tests designed for evaluating children should not be applied to adults. They also should not be used in other assessments, such as gauging mental competency to stand trial.

Further complicating matters is the fact that psychological testing is big business. When publishers can sell not only the test itself, but related materials like answer sheets and scoring programs, for hundreds of dollars, developers might well feel pressured to publish sooner rather than after lengthy review and testing.

As to why the challenge rate is so low, the authors describe several possible contributing factors. Some legal proceedings do not apply the same rules of evidence as others. Lawyers might lack the expertise to gauge a tool's validity, and so defer to experts who might not know or admit the tool's shortcomings. Legal precedents and legislative mandates can also push courts toward favoring certain tools. And many plaintiffs and lawyers lack the time and money to gamble on such challenges when their success rates are so low.

Neal hopes the research helps people realize they can take a second look at psychological tests involved in their legal proceedings.

"If you're interacting with a psychologist in any kind of legal context – if you're being sued in a civil tort for emotional harm, if you are going through a child custody battle – and there's some kind of psychological assessment happening, and there's some tool involved, that might be challengeable. So you could be skeptical, and you can also encourage your attorney to be skeptical."