Can You Make Decisions Without Data?
Why Lawyers Need to Know Basic Stats and UIUX in the Age of AI
It has become common parlance to talk about making data-driven decisions. At most tech-related conferences, there will be one topic on making more data-driven decisions. It usually involves using some sort of AI, machine learning, or computer smarts, and massive amounts of data.
But as legal professionals, who have a crucial role to play in all sorts of decisions, it begs the question: is there any decision we make that does not require data? Would our clients, or anyone for that matter, trust us if our decisions were driven by absolutely no data? The law calls that irrational. A reasonable person acting reasonably (Associated Provincial Picture Houses Ltd v Wednesbury Corporation (1948) 1 KB 223) can ill afford not to use data.
Sure, we might have heuristics, especially heuristics honed over years and decades of experience, but those heuristics need some data to work (FN: there’s an argument here that the superiority of human learning and reasoning lies in the fact that we get things done on far less data than AI). While we talk about the importance of being realistic, objective, reasonable, and sometimes, wise, in all sorts of situations, we usually refer to data only in specific contexts where there is some dimension involving digitalisation.
Somehow, that becomes an intellectual crutch for reasoning about data, that we never really run from. It is a mode of reasoning that eschews the same paths that we tread when we reason about experience and facts – even though they are the same thing. Perhaps the combination of digital and scale and frequency is something that shocks us out of travelling those paths. But it should not.
Perhaps you need to be further persuaded. If the law truly touches on every facet of life, as legal professionals, we need to be clearer about what we mean when we talk about data. For one, the data that sits in our document repositories probably relates to a much wider swathe of human life than the average document repository in the average business. Yes, the legal profession is special that way.
And yet, our preferred term of art when referring to some digital information is just “data”. Should we reason about this qualitatively? That is our preferred mode of reasoning in the legal realm. What about quantitatively? It should not be relegated to the realm of the P&L sheet.
Is it possible to reason about that data both qualitatively and quantitatively? For example, “yes, this set of document data may prove useful in ascertaining certain claim amounts, but do we have enough documents to train an accurate model?”
Gobbledygook?
How about: “Can I give this set of documents to a trainee and expect them to understand how to calculate claim amounts?”
Probably less gobbledygook, but both sentences speak to the same issue.
Our reasoning about data often takes the form of “is the computer smart enough to make use of this data?” As I argued in a previous piece, it’s important to move towards reasoning in the form “We must use X data to do Y“. How does one get to Y?
If the last few years are anything to go by, the journey will involve some form of AI. At its base, AI runs on statistics. What you see in ChatGPT’s chat window, or in Anthropic’s Claude, or in Google’s Gemini is artifice to some extent: gargantuan language models using ridiculously multidimensional (not talking about Marvel here) numbers abstracted behind a friendly chat window. Several New York lawyers have been fooled by this seemingly uncanny combination. It really should not be the case, but perhaps this is exactly the reckoning that the legal industry needs if the profession is to assist society in navigating the travails of our modern world. Part of that reckoning means we need to remember some secondary school statistics.
According to Oxford Reference, data is simply defined as: “A collection of facts or organized information, usually the results of observation, experience, or experiment, or a set of premises from which conclusions may be drawn”. In this piece, I explain several concepts that you are likely to come across.
Whether you are reasoning about whether a sample is important, relevance scores (how does one judge relevance using numbers?), confidence scores (how does one judge confidence using numbers), or outcome probabilities (how does one pin numbers to the likelihood of an outcome), or just what AI is doing, you will need to engage with the following concepts.
1. Defining Population and Sample
Population: The entire set of items or individuals of interest.
Sample: A subset of the population used to make inferences about the whole.
2. Population vs Sample Size
Bottom Line: Drawing valid conclusions requires understanding the difference between population and sample size.
Population: The entire set of coin tosses, akin to a full symphony orchestra. Sample: A subset of these tosses, like a chamber quartet, providing a glimpse into the whole.
Example: A law firm seeking to understand client satisfaction can survey a representative sample rather than every client, gaining valuable insights without the exhaustive effort of surveying the entire population.
3. Law of Large Numbers (LLN):
Bottom Line: Larger samples provide more accurate and reliable insights – usually (this involves a more technical explanation, but briefly, not everything involves the bell curve, as is so common in our Singapore experience).
Imagine tossing a coin. Initially, you might get a streak of heads or tails, but as you toss the coin more times, the proportion of heads and tails will approach 50% each. The Law of Large Numbers tells us that the more coin tosses (data points) we collect, the closer our results will reflect the true probability.
Example: Evaluating the average damages awarded in breach of contract cases is like tossing the coin many times; the more cases you consider, the more accurate your average becomes.
4. Central Limit Theorem (CLT):
Bottom Line: We make some assumptions about how data is distributed (usually, that it’s bell-curved) in order to carry out some basic inferences based on statistics.
Consider each coin toss as part of a larger game. Even if the original data (like the distribution of case outcomes) is not normally distributed – like a bell curve, the Central Limit Theorem states that the distribution of the sample mean will approach a normal distribution as our sample size increases. This holds true even for non-normal distributions, making it possible to infer broader truths from sample data with some confidence.
Example: We may wish to study whether innovation reduces case resolution times. We could take one sample of 50 cases, and observe that the sample’s mean resolution time is 100 days. As we increase the sample size, say from 50-100, and from 100-150, the mean of the sample becomes approximately normal (bell-curved). That does not mean the underlying population’s distribution is normal. However, this does let us make some basic inferences about its statistics, by relying on known properties of the normal distribution. We could compare this sample to another large enough sample, to see if their means differ, and therefore whether innovation made a dent on case resolution times (credit to Asst Prof Jerrold Soh – more details below – for assistance in clarifying the concept).
5. Precision and Recall
Bottom Line: Balance between precision and recall ensures effective and efficient AI systems.
Precision: Picture a coin that lands on heads 80% of the time when a system predicts heads. Precision measures how often the system’s predictions are correct.
Example: In document review, if the system flags 100 documents as relevant and 80 truly are, the precision is 80%.
Recall: Now imagine the system aims to predict every occurrence of heads. Recall measures how many times the system successfully predicts heads when it does occur.
Example: If there are 100 relevant documents and the system identifies 80, the recall is 80%.
Balancing precision and recall has been a perennial issue in search, and now, AI systems. Lawyers familiar with sensitivity and specificity in eDiscovery will no doubt find these concepts familiar.
6. Matrix Multiplication
Bottom Line: Understanding matrix multiplication is essential for complex data transformations in AI algorithms.
Matrix multiplication is similar to combining multiple outcomes of coin tosses into a larger, multi-dimensional game. It is a fundamental operation in linear algebra, which is the basis for the sophisticated data-munging in AI algorithms.
Example: In legal analytics, matrix multiplication transforms data sets, enabling the analysis of relationships between multiple variables: not just X against Y. Consider a model predicting case outcomes, where matrices represent various influencing factors, allowing for a multi-dimensional analysis that allows you to get closer to the crux of an issue.
7. Bayes’ Theorem
Bottom Line: Update your assumptions when faced with new evidence.
Imagine you initially believe a coin is fair. After seeing it land heads several times in a row, you update your belief about the coin’s fairness. Bayes’ Theorem is about revising probabilities with new evidence.
Formula: P(A|B) = P(A) P(B|A) / P(B)
Example: We should revise our assumptions about how AI will affect the law when faced with new evidence. Bayes’ Theorem recalibrates the probability based on this new data.
8. Regression Analysis
Bottom Line: Regression analysis helps identify relationships and predict outcomes.
Linear Regression: Imagine plotting the results of coin tosses, against some variable, for example, how strongly the wind is blowing, on a graph to find a trend line. Linear regression models the relationship between a dependent variable and one or more independent variables. It is important to note that models approximate the real world. It’s an intelligent guess, so to speak.
Example: Interested in whether HDB prices will continue to hit the $1 million mark? Real estate analysts likely use regression analysis.
9. Ergodicity
Bottom Line: There is a difference between having multiple trials in the same time period (cross-sectional) and prolonged observation of the same process over time (time-series) (FN: https://stats.stackexchange.com/questions/344937/ergodicity-explained-in-layman-terms). Time is a crucial dimension that is often missed out.
Consider tossing a coin many times over a long period, versus tossing many coins at the same time. Ergodicity implies that the average result over time for one coin, will be the same as the average result for many coins tossed simultaneously.
Example: In legal contexts, ergodicity would mean that case outcomes observed from multiple judges in one time period, should be the same as one judge over a long period of time.
If the coin-toss were a betting game, you could have drastically different outcomes if your win-loss depended on a series of coin tosses, or the combined outcome of many coin-tosses at the same time. In the former, you could be “ruined” if you run of money before you reach your expected outcome (also colloquially known as your “average”). Typical uses of the “average” result often do not consider this dimension.
10. Confidence Level and Statistical Significance
Bottom Line: Confidence levels and statistical significance provide a measure of reliability for your conclusions – but they are only a starting point.
Confidence Level: If you toss a coin 100 times and want to estimate the probability of heads, the confidence level tells you how sure you are about this estimate. A 95% confidence level means you can be 95% certain your estimate falls within a specific range.
Example: When predicting the success rate of cases, a 95% confidence level in your model’s accuracy helps ensure your predictions are reliable.
Statistical Significance: If you find that a coin lands heads 60% of the time, you need to determine if this result is due to chance or if the coin is biased. Statistical significance helps ascertain whether observed effects are genuine.
Example: When assessing whether a new law impacts case outcomes, statistical significance helps estimate the probability that your results are due to random variation.
Why Understand Them?
These statistical concepts are the bedrock upon which machine learning and generative AI stand. If you really want to be driven by data, you need to interpret data with precision and nuance. The bedrock makes it possible.
In practice, this involves:
- Data Collection: Like assembling the pieces of a vast jigsaw puzzle.
- Data Analysis: Employing statistical methods to reveal the underlying picture.
- Modelling and Prediction: Using machine learning models to anticipate outcomes and discern patterns.
- Decision-Making: Leveraging these insights to shape legal strategies, case assessments, business strategies, and client advice.
In the legal domain, this translates into predictive models estimating case success probabilities, natural language processing automating document reviews, or AI-driven tools uncovering hidden trends in legal data. Mastery of these statistical principles enables legal practitioners to actually wield AI with precision, enhancing the clarity and efficacy of their decision-making. After all, as frequently quoted by one of the most accessible writers in our profession: “Any sufficiently advanced technology is indistinguishable from magic” (Arthur C. Clarke)
We do not want to be waving wands.
Underlying the Artifice: Enhancing User Interfaces and User Experiences
This brings us to the intuitive user interfaces and experiences that we have been spoilt by, courtesy of the tech giants. Besides wielding AI technologies with better understanding, how do we also ensure that sophisticated statistical tools and AI technologies are more accessible and usable by all?
The answer is user interfaces that are both intuitive and user-friendly. Here are a few suggestions:
- Simplified Visualisation: Presenting data through clear, interactive visualisations helps users grasp complex statistical insights quickly. What does it mean to have a simple but effective data visualisation? It’s more art than science, and really is akin to effective writing. As legal practitioners, we should gain more exposure to data visualisation techniques, and the stories that they can tell. One of the best authors on the topic is Edward Tufte.
- Enhanced Predictive Tools: Incorporating precision and recall metrics into AI tools ensures they are both accurate and comprehensive. Intuitive interfaces can highlight these metrics, helping users understand the reliability of AI-driven insights. Interfaces can provide such tools when users become more accustomed to them.
For example, a document review system that visually indicates precision and recall scores alongside flagged documents allows users to trust and verify AI recommendations.
- Dynamic Updates: Bayes’ Theorem emphasises the importance of updating beliefs with new evidence. User interfaces can reflect this by dynamically updating predictions and insights as new data is input, keeping users informed and adaptable.
Example: A case management system that adjusts risk assessments and success probabilities as new evidence is added, rather than presenting a static picture. In such a scenario, effective data collection and processing is necessary.
- Confidence and Significance Indicators: User interfaces should clearly indicate confidence levels and statistical significance to help users understand the insights provided. Interfaces must also explain how these measures are reached, in an easily accessible manner, so that users can pursue these insights if necessary.
By marrying sound statistical understanding, with greater familiarity of user interface and experience principles, legal practitioners can not only enhance their practice and decision-making capabilities, but also bring more insight and clarity when advising clients on the statistical problems and algorithms that undergird so much of daily life. Here is to hoping this article got you started.
*This author has had the privilege of working under, and with a number of lawyers who are statistically savvy (including a former boss, and Assistant Professor Jerrold Soh), so this article owes its strengths to them. Its flaws are entirely my own. If you are interested in exploring statistics in the law further, please take a look at Asst Prof Soh’s work:
PHANG, Andrew; GOH, Yihan; and SOH, Jerrold. The development of Singapore Law: A bicentennial retrospective. (2020). Singapore Academy of Law Journal. 32, (1), 804-890. Available at: https://ink.library.smu.edu.sg/sol_research/3156
SOH, Jerrold and GOH, Yihan. How and why do judges cite academics? Evidence from the Singapore High Court. (2022). Asian Journal of Comparative Law. 17, (1), 134-166. Available at: https://ink.library.smu.edu.sg/sol_research/3972
