How science is and isn’t like the legal system

I have been musing a lot lately, motivated in part by the post-fact era we seem to have moved to, on what makes science such a powerful way of knowing. Hopefully, my thinking will advance enough that I can write a post on that soon. The one thing I’m sure of is it is not the conventional answer of “we have the scientific method”. But in the mean time, I served on a criminal jury not too long ago. It got me thinking about how the criminal trail process and science were and were not similar. This seems like at least a good starting point to think about what makes science a powerful way of knowing.

Although I am not legally barred from discussing my experience, probably better not to say too much specific. In general I was on the jury for a case involving OUI (operating a vehicle under the influence – a car and alcohol in this case). It was a very typical trial in the Anglo-Saxon legal system (I am aware many other countries have very different systems and would be interested in hearing how those systems compare to science in the comments). Namely, the state prosecuted, the defendant had a lawyer, the trial started with opening statements, had a presentation of evidence by each side, ended with closing statements and then the jury of 12 went into a room, not to be let out until all 12 of us agreed on guilty or not-guilty.

So similarities:

  1. Belief in one truth – courts and science are among the few modes of human knowing that actually believe there is an objective truth. Ask an English professor what a book means and you’ll get a derisive glance. Theologians generally acknowledge that much of their knowledge is based on belief and subjective to the individual experience. Even social scientists would acknowledge the complexity of a single interpretation. But courts and science are pursuing a more objective sense of truth. I hasten to add that I am not naively claiming science (or the courts) are always right, but they differ from many fields of endeavor which don’t even believe there is a “truth” to find. And both science and the legal system have an objective enough sense of “truth” that they often can test and admit that they were wrong.
  2. Focus on evidence – other than the opening and closing statements, nominally all either side did was introduce evidence to support their side. This included testimony from participants and experts, a good deal of video thanks to the trend to dash-cams and body-cams in the US, and output from a breathalyzer machine. The evidence was carefully enumerated and made available to the jury. The judge repeatedly emphasized it was the jury’s job to interpret the evidence and only the jury could do that job. And one of the few unpardonable sins for a lawyer in pursuing victory is to hide evidence from the other side. The parallels to science with its focus on an evidential approach are obvious.
  3. How about that trial structure – opening statement, presentation of evidence, closing statement. Sound like the structure of a paper you’ve written recently?
  4. Standards of certainty – science for better or worse (probably mostly for the worse) asks for p<0.05. A criminal trial requires “proof beyond a reasonable doubt” (something which is less quantitative but the judge expands on in some detail). A civil trial (a law suit) requires only a preponderance of the evidence. Of course criminal trials also have a presumption of innocence. The burden of proof is entirely on the prosecutor. This is when you get down to it a clear statement of the relative cost of Type I vs Type II error. Trials  and science are both much more Type II accepting (failure to convict, failure to draw a conclusion) to avoid Type I errors (convicting an innocent person, accepting as true a wrong statement).
  5. Adversarial – the legal system explicitly depends on people trying to take different sides and arguing in public with each other for others to judge. Although science likes to fool itself (e.g. that one paper with one experiment decisively decides the truth) the reality is that on the really big questions we argue with each other, often for a long time, before agreeing. And one of the strongest arguments in support of scientific accuracy is that there are incentives to disagree and disprove something others think is true.
  6. Social decision making – Again scientists sometimes like to think that a single experiment reveals the truth. But in reality, science’s version of truth (which is admittedly sometimes wrong) comes from a rather social process of scientists listening to different sides, examining the evidence and deciding what they believe. As Lakatos and then Kuhn highlighted, science is a social process. The jury system makes this explicit with 12 people having to come to agreement.

And some differences:

  1. The adversarial system is rigid – the roles are proscribed – the prosecutor wants to convict, the defendant wants a decision of not guilty. They are not just allowed but expected to bias their presentation, trusting the truth to come out between the adversaries. Science certainly has its sides of partisanship and bias. But these sides are self-imposed and can be abandoned at any time. While a prosecutor should not lie or hide evidence, and should drop a case if they become convinced the defendant is innocent, they wake up in the morning with no choice about which side of the argument they will come down on. In the criminal justice system the advocates are rigidly fixed in their roles and the jurors are rigidly neutral (the process to find a random neutral jury took as long as the trial itself). In science, the advocates are the same people as the jurors. And as a result they have to be willing to be flexible and change their minds. A good scientist shouldn’t have a pre-determined rigid answer to a question.
  2. Lack of investigation – we jurors were told over and over not to investigate the situation ourselves. We were to make our decision only on the basis of the evidence presented to us. I can tell you in the case I was on there were at least two whopping big questions hanging over the case that nearly every juror in the room identified as very important but not addressed by either lawyer. Either one of them (whether the defendant’s schedule allowed time to drink before being stopped in the car, whether a particular medical condition could affect breathalyzer tests) could have changed the outcome. We could have answered one of these with 10 minutes on google and the other with some very simple subpoena of records. But we couldn’t use any of this. Scientists obviously are the opposite – if they need more information, they are expected to go get it before making an opinion.
  3. Reliance on personal testimony – although science and trials share a focus on evidence, trials recognize testimony of individual people under oath as a major form of evidence. They certainly acknowledge the possibility of lying and explicitly instruct jurors to decide what testimony they believe. My case was unusual in that there was so much video footage, but still a majority of the case came down to testimony by the police officers, and most cases even a few years ago would have had only testimony. Science on the other hand, doesn’t accept testimony. Or does it? What else is the methods and results section of a paper? I’m on the fence whether science is so different on this one.
  4. Have to have an answer – I think most of the above differences boil down to one big difference. At the end of a trial a jury is supposed to come out with a verdict.  “We need more research” is not really an acceptable answer in a trial. One can have a hung jury (not getting to 100% agreement), but this is usually looked at as a failure of the system, and the solution is usually a retrial (changing the jury pool), not the pursuit of more information. Science of course has a strong bias towards “we need more information” rather than reaching a wrong conclusion.

And one comparison I am not sure whether to label as a similarity or a difference:

  1. Attitude to external technology – as is probably true in many OUI cases, the breathalyzer results were pivotal (in my trail the jurors all looked at a piece of paper that said the defendant was over the legal limit). I personally found the attitude of my fellow jurors surprising. There was a complete willingness to speculate about humans involved being wrong (either accidentally or on purpose by lying). But there was a number on a piece of paper (over the legal limit) and it was in black and white, and that was it. Humans were dubious, but the machine was infallible. Questions of calibration were not really something people were interested in. Nor was the possibility that a particular medical condition could affect the results. And wow was the expert testimony on the breathalyzer weird to me. Like I presumein  all OUI cases, there was an expert witness who described how the breathalyzer worked and how they’re calibrated and why they’re reliable. I get that there was not time nor probably interest among the jurors about the technical details (it turns out it works by shining a specific frequency of infrared light which is absorbed by alcohol through and looking at the amount of absorption). But to my eyes the expert witness obfuscated and hand waved and didn’t really address any of the important questions (e.g. specificity of the absorption to alcohol vs ketones and other non-illegal chemicals). And there are very quantitative assessments of the accuracy of these breathalyzers in the literature that I found out on my own later that could be put in simple numbers (error bars around estimation of blood alcohol concentration, % correct answers in patients with the medical condition). Short answer quite accurate and fairly robust but not perfect to the disease in question. But the expert obfuscated, belaboring incredibly unimportant points and ignoring important ones, and the jury treated the machine as infallible and completely avoided what most scientists would consider a fairly straight forward approach to assess accuracy and reproducability. This would make me think of this as a difference. But then I’ve observed scientists go gaga over a simulation model, or look at a map full of massive interpolations, or skip over and yet simultaneously completely yield to the correctness of a page of equations, or take data from another discipline like climatology and assign them all very high accuracy and infallibility as well. So I’m not so sure. But it certainly was an eye-opener to me in outreach in how differently scientists and the public think about the question of accuracy.

My net conclusion is that science and law have more similarities to each other than does science with most other fields or modes of human knowledge, but the differences are still pretty substantial. I think the essence of finding what makes science such a powerful way of knowing (and ultimately informing policy) probably lies in both the similarities and the differences.

What do you think? Are the other similarities or differences between science and jury trials that I didn’t identify? How about with the legal system in your country? Is the legal system a close analog to science? If so what does it highlight (in its similarities or differences) about why science is a powerful way of knowing?

5 thoughts on “How science is and isn’t like the legal system

  1. Interesting thoughts.

    Re: the adversarial system, anecdotally it seems like the scientific debates most analogous to the adversarial legal system are the scientific debates *least* likely to lead to scientific progress. Polarized debates between opposing camps, each unable to agree on what would constitute a decisive test of their opposing views, is a recipe for stagnation in science. Think of the pointlessness of the Ehrlich-Simon wager ( I’d be interested in whether there’s comparative historical work, or mathematical modeling work, on whether/how fast different systems converge on the truth.

    It seems like there’s maybe a loose analogy here to evolution. How fast is convergence to an adaptive optimum when you just have evolution by natural selection choosing among different fixed phenotypes, vs. when you also have adaptive plasticity? The former case seems loosely analogous to the adversarial system, the latter loosely analogous to a system in which individuals are free to change their minds and often do so.

    Also, your post reminded me of this old post, which I remain quite proud of even though basically nobody read it or commented, exploring the analogy between different peer review systems and different forms of procedural justice:

  2. That’s a really interesting comparative exercise, Brian, I enjoyed reading it. I can think of one other parallel: “innocent until proven guilty” could be viewed as “hypothesis supported until evidence found to falsify it”. Are there modern legal systems where innocence, rather than guilt, must be proven? Or did we leave that behind in the era of witch trials…..?

    • I believe the English law of defamation still basically requires the defendant to prove the truth of their statements or that one of the other defences applies.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s