TAR Wars Episode IV: The CAL Zone

A rather heated skirmish has broken out in the community of eDiscovery consultants who specialize in search technology.  Originally fostered by an ACEDS seminar held in early August, the conversation has spread to a dispute over definitions of technology, which many observers, myself included, feel will serve only to further hinder acceptance of this technology.

In writing this overview, I am going to paraphrase statements or positions as little as possible, and instead provide direct quotes and links to the original comments of the various people quoted.

On August 1, Judge Andrew Peck released his decision in Hyles v. New York City, No. 10 Civ. 3119 (S.D.N.Y. Aug. 1, 2016), where he refused to order a producing party to use TAR.  Despite stating that he was himself a “judicial advocate” of using TAR, he felt that the 2015 amendments to the Federal Rules of Civil Produce had set a standard of reasonableness for preparing productions; and absent a finding their methodology was unreasonable, he could not order a party to use TAR.

Shortly after the Hyles decision, ACEDS’ Executive Director Mary Mack, moderated a webinar presented by Doug Austin, Bill Dimm and Bill Speros entitled, Faster, Better, Cheaper: How Automation is Revolutionizing eDiscovery.  The webinar was critical of some of the justifications for TAR, including the following statement from Hyles:

To be clear, the Court believes that for most cases today, TAR is the best and most efficient search tool. That is particularly so, according to research studies (cited in Rio Tinto), where the TAR methodology uses continuous active learning (“CAL”), which eliminates issues about the seed set and stabilizing the TAR tool.

It was also critical of a key assumption stated in Da Silva Moore v. Publicis Groupe, which quoted (in dicta) directly and without qualification from a Richmond Journal of Law and Technology (JOLT) article stating that:

“Technology-assisted review can (and does) yield more accurate results than exhaustive manual review.”

The webinar participants felt that this statement in particular was not adequately supported by enough scientific testing, and that TAR did not have quite the uniform unqualified technical support that some courts were according it.

The next day, ACEDS published Dr. Gordon Cormack’s response to the seminar – https://www.aceds.org/news/303311/A-Considered-Response-from-Gordon-Cormack.htm. Dr. Cormack is a leading expert in the field of computers and technology and has published numerous articles and studies in the field, which form the basis for much of the legal support for this technology – including the JOLT article mentioned above. He and Maura Grossman, the equally well known attorney specializing in computerized search technology who has also been involved in much of the leading research in this area, have partnered in developing their own system, for which they have a patent as well as a trademark for the phrase “Continuous Active Learning.”

That dialogue has continued, both on the ACEDS site and on Ralph Losey’s blog (more on that below), and rather than paraphrase them, I’ll suggest you read them yourself.  What is most interesting to me, and a point to which I will return later in this accounting, is Dr. Cormack’s statement at one point, that the webinar was based upon “selective reading” and “presented false impressions.” And further that Bill Speros in particular was presenting a “strawman” argument in his dissents.

I assume here that by “strawman” Dr. Cormack was not referring to the common usage of that phrase, to mean a person being used by another individual, typically to make a purchase, who does not want his identity revealed.  Rather, I believe he meant the logical fallacy of misrepresenting someone’s argument in order to make it easier to refute.

I don’t understand this position, and it is this lack of grasping the real basis of the argument that causes me to say that this entire dispute is serving only to lessen the likelihood of the adaption of CAL. I have noted before that the majority of attorneys are not using it. Surveys show this to be true. And even the Gartner group in its recent Hype Cycle for Emerging Technologies stated that machine learning technology is at the peak of “inflated expectations,” which naturally leads to the “trough of disillusionment.”

Witness that the debate about CAL has continued on Ralph Losey’s blog, with his discussion on September 11 of what he calls Predictive Coding 4.0. – (https://e-discoveryteam.com/2016/09/11/predictive-coding-4-0-nine-key-points-of-legal-document-review-and-an-updated-statement-of-our-workflow-part-one/ ). His new interpretation of best practices in this area to include active human participation was, in my mind, the most instructive. But it was shortly followed on September 14 by blog posts from Dr. Herb Roittblatt – (http://ediscoveryscience.blogspot.com/2016/09/recall-magical-thinking-and-assessment.html) and John Tredennick – (http://catalystsecure.com/blog/2016/09/ask-catalyst-is-recall-a-fair-measure-of-the-validity-of-a-production-response/), which continued the argument that recall measures are not just adequate, but the best measure of how such a system works.

Dr. Roittblatt went so far as to say that since we can’t measure quality, of necessity, we must stand behind measure of quantity.  Spoken like the true scientist he is, but I would argue that we, as lawyers and judges, can and do measure quality. It’s what we do. We don’t use just numbers for that measure.  We use concepts, like relevant and probative.

And now I come to a fine interview with Maura Grossman, on September 16 in Artificial Lawyer, a periodical dedicated to AI and the Law called AI and the Future of E-Discovery: AL Interview with Maura Grossman. In that article, Maura made several points I find relevant to our discussion. Once again though, I caution that these are excerpts, and you should read the entire article to see if you draw the same conclusions as I do.

Maura said, among other things:

“. . . many vendors and service providers were quick to label their existing software solutions as “TAR,” without providing any evidence that they were effective or efficient. Many overpromised, overcharged, and underdelivered. Sadly, the net result was a hype cycle with its peak of inflated expectations and its trough of disillusionment. E-discovery is still far too inefficient and costly, either because ineffective so-called “TAR tools” are being used, or because, having observed the ineffectiveness of these tools, consumers have reverted back to the stone-age methods of keyword culling and manual review.”

I’ll come back to this at the end, because I think one of her points here is the exact embodiment of the problem we face in today’s technology market.

But before that, let’s return to Ralph Losey and his blog post of September 18, where he revisits his explanation of his Predictive Coding 4.0 model – https://e-discoveryteam.com/2016/09/11/predictive-coding-4-0-nine-key-points-of-legal-document-review-and-an-updated-statement-of-our-workflow-part-one/#comment-184458

In that article, Ralph says:

“Our method is Multimodal in that it uses all kinds of document search tools. Although we emphasize active machine learning, we do not rely on that method alone. Our method is also Hybrid in that we use both machine judgments and human (lawyer) judgments. Moreover, in our method the lawyer is always in charge. We may take our hand off the wheel and let the machine drive for a while, but under our versions of Predictive Coding, we watch carefully. We remain ready to take over at a moment’s notice. We do not rely on one brain to the exclusion of another”

“For us supervised learning means that the human attorney has an active role in the process. A role where the attorney trainer learns by observing the trainee, the AI in creation. I want to know as much as possible, so long as it does not slow me down significantly.”

“In other methods of using predictive coding that we have used or seen described the only role of the human trainer is to say yes or no as to the relevance of a document. The decision as to what documents to select for training has already been predetermined. Typically it is the highest ranked documents, but sometimes also some mid-ranked “uncertain documents” or some “random documents” are added in the mix. The attorney has no say in what documents to look at. They are all fed to him or her according to predetermined rules. These decision making rules are set in advance and do not change. These active machine learning methods work, but they are slow, and less precise, not to mention boring as hell.”

But the biggest problem I see here, is that we have experts in our field disagreeing as to the best standard for CAL.  As Maura said in her Artificial Lawyer interview:

“Not all products with a “TAR” label are equally effective or efficient. There is no Consumer Reports or Underwriters Laboratories (“UL”) that evaluates TAR systems. . . . if they try one TAR tool and find it to be unsatisfactory, they should keep evaluating tools until they find one that works well.”

But who has the time, or budget, to do that?  In the absence of a standards body, we look to experts to advise us; and if the experts all disagree, especially in highly technical disputes, then the end result is that people will simply not use the technology.

In his comment to the Ralph Losey discussion, Craig Ball said:

“Any affordable technology that pulls away as clearly superior to its alternatives in performing a particular task is going to establish a de facto standard of care when examining the reasonableness of conduct.

This ongoing debate shows we are nowhere near that “clearly superior,” de facto standard. As Judge Peck said in the Hyles decision:

There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.”

Indeed.

Tom O'Connor

TOM O’CONNOR

Senior eDiscovery Consultant
Advanced Discovery

Tom O’Connor is nationally known as a consultant, speaker and writer. In his consulting work, Tom specializes in working with law firms and their corporate clients to manage their electronic discovery needs. As senior eDiscovery consultant for Advanced Discovery, he assists clients in database design, implementation and training. He also addresses specific eDiscovery issues such as retention policies, litigation holds and document exchange protocols. Tom speaks at major legal conferences and writes for both the Advanced Discovery blog and national legal publications.

Send to a friend