An August 5, 2020 webinar organized by the University of Miami received a great deal of attention for remarks made by the National Conference of Bar Examiners (NCBE) president Judith Gunderson suggesting that NCBE critics might encounter difficulties with their character and fitness evaluations. I was equally struck, though, by a remark made later in the webinar by David C. Reeves, the chair of the Florida Board of Bar Examiners: “It is unwise to make wholesale changes, in my view, to systems, to processes while we’re in the middle of the pandemic that has led to the need for short-term accommodations.” I disagree. A crisis is the perfect time to make wholesale changes to systems and processes. Hole in the ozone layer? Time to ban CFCs. World war depleting the male workforce? Send in the women. Mortgage crisis? Revise subprime lending rules. #Barpocalypse? Rethink attorney licensure.

There has been a lot of recent debate on whether attorney licensure exams protect the public. That is an empirical question calling for a data-driven approach. As a bona fide data nerd, I have accepted the challenge. If the bar exam and Multistate Professional Responsibility Exam (MPRE) serve their stated purposes, then we should see a direct correlation between ethical violations and MPRE cut scores and an inverse correlation between ethical violations and bar exam pass rates. In short, if the tests protect the public, the more test-takers they weed out, the more we should see complaints and charges against attorneys go down. Additionally, if competence correlates with a reduction in ethical violations, we should observe the number of complaints and sanctions per attorney go down as experience increases.

There is no correlation between ethical violation rates and licensure exam stringency, nor is there a correlation between ethical violation rates and years of experience.

Evidence from Illinois

Illinois’ Attorney Registration and Disciplinary Commission (ARDC) provides an unusually rich and detailed array of data in its annual reports, making it a fruitful investigatory starting point. The ARDC annual report includes demographic data on attorneys subject to disciplinary investigations, disciplinary charges filed, and sanctions imposed. The numbers indicate that failure to provide competent representation is not a major source of complaints or discipline and new attorneys are far less likely to commit ethical violations than experienced attorneys.

• From 2011 to 2018, only 1,773 (3.9%) of the 45,823 investigations initiated by the ARDC were for failure to provide competent representation.
• Of the 759 charges filed before the Hearing Board in that time span, only 43 (5.7%) were for failure to provide competent representation, only 27 (3.6%) were filed against early-career attorneys (those in practice under five years), and only 3 (0.4%) were filed against an attorney 21-29 years of age.
• Of the 943 attorneys ultimately sanctioned by the Illinois Supreme Court in those eight years, only 14 were early-career and only 2 were under the age of 29.
• Early-career attorneys made up an average 12.5% of the attorney population but accounted for only an average 4.1% of investigations and 2.8% of sanctions.

Investigations

Case Filings

Sanctions

The evidence from Illinois suggests that (a) the profession has more of an integrity problem than a competence problem, and (b) when it comes to public protection, we should worry more about the experienced attorneys than the new ones.

“Silly rabbit,” you may be thinking, “the reason so few ethics complaints stem from incompetence is that we weeded out all the incompetent lawyers with the bar exam.” Dive down the rabbit hole with me and you will see that is not true.

There is no relationship between bar exam pass rates and ethical violations. Similarly, there is no relationship between MPRE cut scores and ethical violations. Either the tests are not doing what they are supposed to do or we have no good metric for “public protection.” It is likely that both are true: the tests do not protect the public and we lack a good metric for “public protection.”

Neither the Bar Exam nor the MPRE Demonstrably Protect Against Ethical Violations

I aggregated information on 2014-2018 bar passage rates, MPRE cut scores, complaints against attorneys, and disciplinary sanctions for all fifty states and Washington, DC from data made publicly available by NCBE, the ABA Center for Professional Responsibility, and the State Bar of California. The final analysis did not include data from 2017 because I could not find an ABA report for that year. The analysis only includes data points in each year for states in which complete information was available; however, each analysis still included observations of at least forty states.

Recall that if the exams have a protective function, we should see some correlation between bar passage rates and measures of ethical violations and/or between MPRE cut scores and measures of ethical violations. The measures of ethical violations in my analysis are a complaint index (number of complaints in each state divided by the number of practicing attorneys in that state) and sanctions index (the number of private and public sanctions in each state divided by the number of practicing attorneys in that state). The charts below show that when states are aligned in order of ascending bar passage rates or MPRE cut scores, the patterns in the complaint and sanction indices are totally random.

We also need to examine averages over time since the pool of lawyers who pass the MPRE and bar exam each year is different from the pool of practicing lawyers that year. The 2014-2018 average metrics show the same lack of correlation.

We Have No Good Metric for Public Protection

Absence of evidence is not, of course, evidence of absence. It is entirely possible that incompetent representation is underreported. It is possible that if reporting mechanisms were altered, we would see some correlation between ethics sanctions and the stringency of licensure requirements. In fact, a 2018 ABA report asserted that disciplinary systems across the country are poorly devised and underinclusive.

The existing system of regulating the profession is narrowly focused on violations of professional ethics. It provides no mechanisms to handle other types of clients’ complaints. The system does not address complaints that the lawyer’s service was overpriced or unreasonably slow. The system does not usually address complaints of incompetence or negligence except where the conduct was egregious or repeated. It does not address complaints that the lawyer promised services that were not performed or billed for services that were not authorized.

Some jurisdictions dismiss up to ninety percent of all complaints. Most are dismissed because the conduct alleged does not violate the rules of professional conduct. The Commission has gathered much information about these dismissed complaints. It convinces us that many of them do state legitimate grounds for client dissatisfaction. The disciplinary system does not address these tens of thousands of complaints annually. The public is left with no practical remedy. While some states have created fee arbitration and other programs, additional avenues should be created in all states to resolve these complaints.

The disciplinary process also does nothing to improve the inadequate legal or office management skills that cause many of these complaints. Many state bar associations have mandatory continuing legal education, substance abuse counseling, and other programs. However, these programs usually are not coordinated with the disciplinary process. Lawyers with substandard skills often need more help than these programs can provide. The judiciary and profession should create new programs and coordinate all such programs with the disciplinary system.

If, however, incompetence is underreported, all that means is that numerous incompetent attorneys have passed the bar and the states are failing to adequately protect the public from them. If numerous incompetent attorneys pass the bar, though, it does not follow that those who failed must be even more incompetent. The exam simply does not measure competence.

The Dim-Recall-of-Minutiae Problem

“Minimal competence” is a slippery beast. Am I really minimally competent to practice if, after two months of post-law-school study, I can regurgitate the rule that if a purchase money secured party perfects within twenty days of the debtor taking possession of the collateral, the purchase money security interest takes priority over a lien that arose during the twenty-day period? Am I minimally competent because I can afford two months out of work to memorize thousands of similar oversimplified, yet inordinately complex, rules across sixteen areas of practice, most of which I will immediately forget after the exam? A majority of Unified Bar Exam states have a cut score of 266 or 270 out of 400. That is 66.5% or 67.5%: a D. A D-average at my alma mater would get you expelled. According to bar enthusiasts, however, you are minimally competent to practice if you can score a D across sixteen areas of practice: Agency, Civil Procedure, Conflict of Laws, Constitutional Law, Contracts and Sales, Corporations, Criminal Law, Criminal Procedure, Evidence, Family Law, Partnerships, Real Property, Secured Transactions, Torts, Trusts, and Wills and Decedents’ Estates. A licensure exam that truly tests minimal competence would either look like the Multistate Performance Test, as many have argued, or require B-level recall of the broad strokes of multiple practice areas rather than D-level recall of the minutiae.

The MPRE suffers the same dim-recall-of-minutiae problem. The cut score for MPRE passage ranges from 75 to 86, depending on the jurisdiction, out of a total 150 possible points. That means to pass the exam in the most demanding jurisdiction, you only need to get about 57% of the questions right. Most people prepare by cramming intricate ethics rules for a few days before the exam. This process—cramming for a few days then achieving what is essentially an F to F+ on an exam—does not make us more ethical. And even if we retained an A-level recall of the rules we crammed, memorizing ethics rules does not induce lawyers to behave ethically. It is hard to see the point of the MPRE from a professional-standards standpoint.

Where’s the Proof of Validity?

None of the analysis above definitively proves that the bar exam and MPRE do not enforce useful professional standards but I have provided substantial empirical evidence that they do not, while no one has presented empirical evidence that they do. Further, the burden of proof should be on NCBE, which has a monopoly on attorney licensure exams and more than $100 million in cash reserves, to demonstrate that the exams protect the public. After everything 2020 bar applicants have endured, they deserve some proof.

The full data used in the analysis presented in this article is available in a GitHub Repository.

JURIST carries extended coverage of Bars Exams in the Pandemic.

Beth Karp is a 2020 law school graduate with 1¾ master’s degrees. She serves as the Director of Accommodations for the National Disabled Law Students Association.

Suggested citation: Beth Karp, There Is No Ascertainable Proof that the Bar Exam and the MPRE Protect the Public, JURIST – Professional Commentary, August 28, 2020, https://www.jurist.org/commentary/2020/08/beth-karp-bar-exam-mpre-data/.

This article was prepared for publication by Tim Zubizarreta, JURIST’s Managing Editor. Please direct any questions or comments to him at commentary@jurist.org