Algorithms have a substantial impact on everything happening around us. Most everyone would agree that the impact of algorithms, computational technologies, and artificial intelligence on everyday life, institutions, and society will only grow, and rapidly. Yet most law students and lawyers lack the foundational knowledge to generally explain how these technologies work, much less assess them.

One way to address this is to teach law students about artificial intelligence and computational technologies. I’m teaching several courses in this arena at Northwestern Pritzker School of Law this year, beginning with “Artificial Intelligence and Legal Reasoning” this fall.

Each class meeting includes discussion about the effectiveness of these technologies, in the proper context of the status quo, and related ethical questions. Additionally, we will spend two full class meetings discussing technical assessment (recall, precision, F1 score, etc.) and broader questions about the use of algorithms, computational technologies, and artificial intelligence in legal-services delivery and society (transparency, explainability, auditability, provenance, bias, fairness, etc.).

I had the opportunity to introduce these topics during multiple talks this summer, beginning with “Demystifying and Assessing Artificial Intelligence” at the launch of the LawAhead Hub think tank at IE Law School in Madrid. The event attracted a variety of leaders, primarily from international law firms and the legal departments of major corporations.

The body of literature on assessing artificial intelligence is developing and growing rapidly. To frame this particular discussion in Madrid, I identified and discussed five resources:

After presenting an overview of rules-driven and data-driven artificial intelligence, I posed the following concepts and questions to generate discussion, primarily from the perspective of evaluating technologies used for the delivery of legal services. The following concepts and questions are mostly gleaned from the Hildebrandt and Goodfellow resources and are not intended to be comprehensive.

Assessing AI (Incomplete Draft for Discussion)

1. What level of performance is required?

  • Who decides whether the machine gets it right?

  • Precision versus recall tradeoffs?

2. Performance on training data? Test data? Segmentation of training & test data?

3. Source of training data?

  • Amount? Quality? Noisy? Right inputs?

  • How does training data differ from “live” environment?

    • Representative? Gaps? Biased sample? Biased labels?

4. How much future data is needed to improve performance?

  • Plot relationship between training set size and performance.

  • How much will it cost to collect and clean the necessary data?

5. Explainability; transparency; auditability?

  • Do we know why certain inputs generate certain outputs?


In addition to taking a deeper dive into these concepts and questions in my “Artificial Intelligence and Legal Reasoning” class, we as a class aim to develop a framework for assessing artificial intelligence as used for legal-services delivery. Using this framework as a lens, we will take a closer look at various categories of legal technology providers.

Twenty years from today these law students will be at the mid-point of their careers. Even in the largest international law firms, where the majority of my Northwestern students will work after graduation, practice will look much different than it does today. Learning about these technologies and how to assess them, including understanding how they will affect legal-services delivery in the future, will serve today’s law students well in their careers. Lawyers with this knowledge and experience will be able to participate in discussions about which technologies to invest in and why, not to mention the opportunities to help build and train AI systems.

Just as important, understanding and assessing computational technologies are critical competencies for anyone who cares about reducing bias and discrimination in the world, increasing access to legal services, increasing justice, and preserving and expanding the rule of law.

Several large law firms and legal technology providers have expressed interest in and support for the development of a framework for assessing artificial intelligence as used in legal technology solutions. My students will work with some of them as they move this project forward. This truly is a class project, so the students will determine its direction, with my guidance and coaching along the way. I look forward to seeing what we develop.

#90MinuteBlogPost