Integreon has an interesting discussion on a recent study pitting humans against machines. No this isn't about supercomputers and Jeopardy! It's something much practical:
The underlying study by a trio of recognized experts in cognitive science, information management, and e-discovery, Herb Roitblat, Anne Kershaw, and Patrick Oot, is described in detail in their journal article, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, published in the January 2010 issue of theJournal of the American Society for Information Science and Technology [link available at the Posse List].
It raises - and partially answers - the important question whether we are approaching a breakthrough in terms of the capability of automated review tools to render ‘consistent’ and ‘correct’ decisions, as measured against an existing standard, while classifying documents in a legal discovery context. The study pitted two teams of contract attorneys against two commercial electronic discovery applications to review a limited set - 5,000 documents - culled from a collection of 1.6 million documents. The larger collection had been reviewed two years earlier by attorney teams in connection with a Second Request relating to Verizon’s acquisition of MCI. The authors’ hypothesis was that “the rate of agreement between two independent [teams of] reviewers of the same documents will be equal to or less than the agreement between a computer-aided system and the original review.”
The study set out to test whether an automated review tool would show similar levels of agreement with classifications made by the original reviewers as did the two contract teams. The two re-review teams agreed with the original review on about 75% of document classification decisions; the commercial automated applications fared slightly better.
There a number of obvious (and not so obvious) flaws in the study, which the Integreon post nicely lays out. My first reaction is that "rate of agreement" is a lousy benchmark, since the measure conflates too many significant variables.
I'm also fascinated by this quest for the document review holy grail: total automation. Contrary to lean principles, these managers seek to automate the process without fully understanding how it works manually. Just exactly how and why do review document reviewers make different calls?
And what about a hybrid approach?
A potential hybrid model would have senior attorneys review representative sets of documents and the tool analyze features of the reviewed documents to identify and auto-tag “like” documents in the larger collection. As the review proceeded, the tool would ‘percolate’ to the review team’s attention subsets of documents from the collection dissimilar from those already reviewed. Based on the reviewers’ decisions as to these documents, the tool continues to apply tags to more of the collection.
The attraction of this approach is two-fold: human attorneys are still making initial determinations but the application magnifies the effect of their determinations by propagating decisions to similar documents throughout the larger collection. It has been suggested that, in the proper context, this approach would permit a single attorney to “review” a vast collection of documents in several hours. A test of that claim is warranted and, if the premise were proved, it would be impressive and could directly influence the increased use of automation in review, even if, for all the reasons stated above, wide adoption of such processes would take a while.
As a lawyer who likes to tightly control processes, I'll admit the attraction of this approach. As one moves down the hierarchy in any litigation team, deep knowledge of the client and issues is inevitable lost. If technology can leverage the knowledge of the most engaged, the better the result, theoretically.
(cross-posted at California E-Discovery Law)
