Skip to main content


Technology-Assisted Review Disputes: New York Pilot Rule Provides National Guidance


In October 2011, the Southern District of New York adopted the “Pilot Project Regarding Case Management Techniques for Complex Civil Cases” (the Pilot Project).[1] The purpose of the Pilot Project is to encourage judges to utilize “best practices” in managing complex civil litigation and to evaluate the efficacy of the management strategies employed. One of its most useful components is the “Joint Electronic Discovery Submission and Order” (the Joint Order). The Judicial Improvements Committee recognized that many complex cases are characterized by the need to take discovery of electronically stored information (ESI); indeed, e-discovery is often what makes some cases complex in the first place.

The Joint Order is designed to focus counsel’s attention on critical e-discovery issues and to identify for the judge potential areas of dispute. Unlike the Model Order Regarding E-Discovery in Patent Cases adopted by the Federal Circuit, the Joint Order does not impose substantive limitations on the scope, form or volume of electronic discovery. Rather, it provides information to assist the judge in guiding discovery on a case-by-case basis, which may include imposing specific limitations, organizing discovery by phases, and utilizing sampling of ESI.

After soliciting basic information about the nature and value of the case, the Joint Order requires counsel to certify that they are themselves knowledgeable about their clients’ technological systems or have involved other persons who have such competency. This provision is intended to avoid the situation where litigation is stymied or becomes unreasonably costly because, for example, counsel are not competent to deal with issues arising from even their own client’s technology.

Next comes the heart of the Joint Order. Counsel identify unresolved e-discovery issues relating to specified subjects: preservation, search and review, sources of production, form of production, identification or logging of privileged material, inadvertent production and cost allocation. Counsel are required to indicate all ESI disputes and set forth their respective positions.[2]

Many counsel are unfamiliar with the Joint Order. They have a difficult time filling it out and are unable to locate quality samples. The Joint Order, however, is not for every case. In the simple case, where there is little or no ESI, preparing the form would be unnecessary. In the complex cases for which it was designed, however, it will be a major asset. While it is required only in complex cases filed in the Southern District of New York, counsel should consider using the Joint Order or a document predicated upon it in the New York trial courts, which would, likewise, benefit from it as a tool for rationalizing complex e-discovery disputes.

In the hypothetical that the panelists utilized at LegalTech in 2013, the primary dispute concerned a party’s proposal to rely upon technology-assisted review for the purposes of collecting and producing ESI. Because the parties had been previously required to set forth their positions in the Joint Order, their arguments at the mock court conference were well-developed. Further, as a result of the Joint Order, the mock court had the benefit of a preview of the issues and the judge was thus able to come to the conference armed with questions that would elicit information addressed to his or her specific concerns. As a consequence, the proceeding was significantly more productive than the typical initial conference where the court is underprepared and the discussion with counsel is often disjointed.


United States Magistrate Judge Lisa Margaret Smith of the Southern District of New York at the New York Predicative Coding Thought Leadership Series held on September 9, 2013, noted, based on her review of decisional authority, the apparent infrequent use of technology-assisted review platforms to search for responsive ESI. She, along with other judges, seek to educate the Bar concerning its pros and cons.

Set out below is an abbreviated version of a hypothetical Joint Order that may serve as a guide when completing an actual Joint Order that contemplates the use of a technology-assisted review platform. It posits a somewhat common trade dress and restrictive covenant dispute.

The parties in the hypothetical cannot agree whether to use a technology-assisted review platform to collect and review ESI or the more commonly used iterative “keyword” search approach. The parties’ responses to the areas that are required by the Joint Order are noted below, as well as commentary on the issues raised by the Joint Order.[3]

Hypothetical Joint Order

One or more of the parties to this litigation have indicated that they believe that relevant information may exist or be stored in electronic format, and that this content is potentially responsive to current or anticipated discovery requests. This Joint Submission and [Proposed] Order (and any subsequent ones) shall be the governing document(s) by which the parties and the Court manage the e-discovery process in this action. The parties and the Court recognize that this first Joint Electronic Discovery Submission No. 1 and [Proposed] Order is based on facts and circumstances as they are currently known to each party, that the electronic discovery process is iterative, and that additions and modifications to this Submission may become necessary as more information becomes known to the parties.

Brief Joint Statement Describing the Action

Plaintiff Victory (Victory), a pharmaceutical company, is asserting that Defendant Sam King (Sam King), in 2013, infringed several well-known trademarks relating to the shape, labeling and coloration of a bottle used for a certain over-the-counter drug. Victory also asserts a claim for the wrongful misappropriation of trade secrets by former employees who left Victory to join Sam King. In addition, there are claims that former Victory employees breached restrictive covenants contained in their employment agreements by joining Sam King.

(a)    Estimated amount of Plaintiff’s Claims:

          x             Between $1,000,000 and $49,999,999

(b)    Estimated amount of Defendant(s)’ Counterclaim/Cross-Claims:

          x             Other (if so, specify) Defendant Sam King Plastics is not asserting any Counterclaims or Cross-Claims.

The description of the underlying case and the amount in controversy seek to frame the issues up front to assist the court to be able to make more informed decisions and to be able to apply the concept of proportionality of ESI expenses to help resolve e-discovery issues.


Counsel certify that they are sufficiently knowledgeable in matters relating to their clients’ technological systems to discuss competently issues relating to electronic discovery, and have involved someone competent to address these issues on their behalf.

This is one of the most important aspects of the Joint Order which requires counsel to affirmatively certify that they have taken steps at an early point in the action to understand their respective clients’ technological systems. Such certification should and needs to result in counsel learning their client’s own technological systems and having to ask the hard questions of their client early on. This certification provides for issues to be presumptively joined on potentially problematic ESI disputes early on so that they can be properly addressed and/or resolved before the action proceeds too far.

Meet and Confer

Pursuant to Federal Rules of Civil Procedure 26(f) (Fed. R. Civ. P.), counsel are required to meet and confer regarding certain matters relating to e-discovery before the Initial Pretrial Conference (the Rule 16 Conference). Counsel for Victory and Sam King hereby certify that they have met and conferred to discuss these issues.

Counsel and their experts have conducted three meet-and-confer sessions at which e-discovery issue were addressed, on the following dates: December __, 2013; January _, 2014; and February _, 2014.

If the parties have their first “meet and confer” just prior to filing the Joint Order, it will likely become clear to the court that they really have done little more than have had a drive-by meet and confer. This requirement mandates that counsel and their clients take their roles in the e-discovery process seriously and work together in advance of the initial conference to discuss matters and to attempt to resolve as many e-discovery issues as possible.

Unresolved Issues

After the meet-and-confer conference(s) taking place on the aforementioned date(s), the following issues remain outstanding and/or require court intervention:[4]

 x     Preservation;

 x     Search and Review;

__    Source(s) of Production;

__    Form(s) of Production;

__    Identification or Logging of Privileged Material;

__    Inadvertent Production of Privileged Material;

__    Cost Allocation; and/or

__    Other (if so, specify) __________________________

This checklist serves as a balance sheet and provides the court and the parties with a snap shot, as of a particular date, of open issues that may require judicial intervention, as well as issues where the parties are in agreement.


1.       The parties have discussed the obligation to preserve potentially relevant electronically stored information and agree to the following scope and methods for preservation, including but not limited to: retention of electronic data and implementation of a data preservation plan; identification of potentially relevant data; disclosure of the programs and manner in which the data is maintained; identification of computer system(s) utilized; and identification of the individual(s) responsible for data preservation, etc.

Victory plans to use technology-assisted review to identify potentially relevant ESI from within the collection set and to preserve all such ESI throughout the duration of the litigation. ESI from within the collection set that are not identified as potentially relevant will be discarded after three months, unless otherwise reasonably requested by Sam King. All original ESI existing on Victory’s systems, regardless of whether or not collected, will be discarded in accordance with Victory’s standard electronic document retention program, which provides for such ESI to deleted after six months. Sam King wants Victory to preserve the entire collection set, as well as any original ESI, for the duration of the litigation.

Preservation is often one of the thorniest issues in e-discovery. It is imperative that the parties agree or agree to disagree as early as possible in a litigation, and to immediately raise the preservation issue with the court for early resolution.


Victory has identified as potential custodians all employees, including management, who were directly involved in the marketing and design of the shape, labeling and coloration of the bottle. Each individual was interviewed to identify potential sources of ESI, which generally consists of email stored both locally and on the corporate exchange server, personal hard drives and network shares, as well as Microsoft Sharepoint data containing collaborative data created by multiple authors. Rather than have these employees collect such data, Victory has engaged a third party vendor to obtain a forensically sound copy of the ESI with the chain of custody of the ESI captured. Mirrored copies of the ESI are maintained by the third party vendor. The date range was restricted to beginning with the initial design concept of the bottle. Collection excluded system and program files which contain no unique data, but are software tools used by the company, as well as obvious spam (as detected by a commercial spam filter). From this collection, all text-based ESI, including email accounts and social media (excluding images that cannot be meaningfully OCR’d or copied in a way that the words in the document are not readable by the review software), Windows and Internet Temp files, and Javascript would be loaded into the technology-assisted review platform for culling.


Sam King has identified its custodians, which include its employees and members of management who were directly involved in the marketing and design of the shape, labeling and coloration of the bottle. Custodians also include the members of management who left Victory and who now work for Sam King as well as the Sam King employees who communicated with the former Victory employees before they joined Sam King. All such individuals are considered custodians, and given the number of custodians and the volume of ESI, Sam King has likewise engaged a third party vendor to collect ESI, based largely on witness interviews to better target the collection efforts. Sam King has generally followed the same process employed by Victory for identifying and preserving ESI.

Each section of the Joint Order contains an overview of the purpose of the section and it sets out the range of issues the parties were to have discussed before coming to the positions reflected in the Joint Order.

2.       State the extent to which the parties have disclosed or have agreed to disclose the dates, contents, and/or recipients of litigation hold communications.

The parties have discussed the use of litigation holds. They have agreed to disclose the recipients of each party’s litigation hold and the dates of the litigation hold. The contents of each litigation hold have not been disclosed.

Parties sometimes do agree on certain items and included here is an agreement on a consensual litigation hold approach.

The Joint Order provides a means to show the court that the parties can agree to some common understandings instead of arguing over every issue. If the Joint Order indicates a disagreement on every issue, it will be clear to the court that the parties are not trying to solve their e-discovery issues.

3.       The parties anticipate the need for judicial intervention regarding the following issues concerning the duty to preserve, the scope, or the method(s) of preserving electronically stored information:

The parties have agreed on preservation generally, but still have disputes on the appropriate method of culling and searching for ESI. If technology-assisted review is used, the structure of the search will be impacted by what is loaded into a coding platform. The value of keyword searching, on the other hand, will be driven by the selection of keywords. The parties are deferring to a future date to decide what types of ESI to include, such as Microsoft Word, PST files for email, and Adobe Acrobat pdf files, and will work cooperatively on this stage of the process.

Search and Review

•    The parties have discussed methodologies or protocols for the search and review of electronically stored information, as well as the disclosure of techniques to be used.

•   The parties disagree on the proper protocol to be used to identify potentially relevant documents for privilege and responsiveness review. Victory believes both parties should use technology-assisted review, while Sam King believes an iterative keyword search process should be used with manual review. Victory proposes the use of technology-assisted review using the following protocol.  Victory submits that it will provide transparency and confidence to Sam King and the court in how the process of identifying responsive documents would be accomplished.  Victory seeks to use technology-assisted review to save money and to improve the accuracy of the review. Victory submits that studies have demonstrated that human reviewers are less accurate in identifying responsive documents than technology-assisted review.  As set forth below, to date, the parties have addressed the following issues:


Victory proposes the use of technology-assisted review using the following protocol.

•    Load all text-based documents from the collection set.

•    Code the training set relying on a senior lawyer who is knowledgeable about the facts of the case using random selection.

•    Log Victory’s obvious privileged and sensitive documents.

•    Review of the training set by counsel for the opposing party.

     – Limit review of the training set to Sam King’s outside counsel.

     – Disagreements as to training set are to be resolved by judicial intervention.

•    Implement revisions to the training set as per agreement or court order.

•    Implement technology-assisted review.

•    Undertake validation of results.

                          – Review 385 documents from the relevant set as predicted by the coding software (the Responsive Validation Set).[5]  In addition, review 385 documents from the non-relevant set as predicted by the coding software (the Non-Responsive Validation Set).  The sample sizes were calculated to achieve a 95% Confidence Level ± and a 5% Confidence Interval of ± 5%.[6]

          – Achieve recall of at least 75% of the responsive documents in the collection and no precision target.[7]

•    Identify and log the privileged and sensitive documents from the Responsive Validation Set and the Non-Responsive Validation Set.

•    Opposing counsel reviews the Responsive Validation Set and the Non-Responsive Validation Set which does not contain privileged or sensitive documents.

•    Implement additional coding, if necessary, if the parties determine additional training of the coding software is required because the recall rate does not reach 75%.

Victory submits that Sam King should be required to follow the same protocol and opposes the keyword search protocol proposed by Sam King. Victory submits that, if the court is inclined to permit the proposed keyword protocol, Sam King should be obligated to undertake the same validation process outlined above with respect to keyword searching to determine whether Sam King’s review process was effective and achieved the same recall targets.

This means that Sam King would need to demonstrate that the keywords used to search for responsive Victory ESI actually caused Sam King to find 75% of the responsive documents in the collection set. This would need to be verified by comparing a random sample of 385 documents from documents found by keyword searches identified as relevant after a manual review with a random sample of 385 documents not containing any keywords which were subject to no additional review for responsiveness.

Victory disagrees with Sam King that keyword searching results should be subject to less scrutiny and validation than technology-assisted review results.  Victory submits that this would penalize it for being progressive and using an advanced technology which studies suggest is faster, cheaper and more accurate than keyword searching and it would act as a deterrent to improving the discovery process.


Sam King opposes the use of technology-assisted review, which it claims is an unproven technology for the effective identification of relevant documents for purposes of a production in litigation.  Sam King believes putting human eyes on every document as a means to provide reasonable assurances that an effective search was conducted is the appropriate methodology to identify responsive information.  Sam King does not believe that proportionality of expense concerns should require the use of technology-assisted review.  If the court is inclined to permit technology-assisted review, Sam King submits that Victory should be obligated to work with their technology tools to achieve a “recall” rate not of 75%, as suggested by Victory, but of 90% or better.

Sam King submits that the use of its proposed keyword search protocol follows The Sedona Conference® Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, dated August, 2007, by using an “iterative” approach based on creating keywords, sampling the hits, and using this feedback to revisit the keywords and improve the keyword list. This industry norm is an approach used in the vast majority of cases. Sam King recognizes that The Sedona Conference® is updating their Commentary, but, at the present time, keyword searching is commonly employed by lawyers in this type of case. Sam King proposes using off-shore reviewers to perform a first pass review to control and reduce the costs of the review.

Sam King does not believe that the Non-Responsive Validation Set or any non-responsive documents should be shared with Victory in any instance, even in small sample sets, which is one reason it is concerned about using technology-assisted review coding at all despite potential cost savings.  If human review is undertaken, there is no need for such intrusive review or whether recall rates or targets are met because a lawyer reviews every document which hit a keyword. Sam King submits that technology-assisted review adds a layer of intrusion and complexity into the discovery process which it is uncomfortable agreeing to.

1.       The parties anticipate the need for judicial intervention regarding the following issues concerning the search and review of electronically stored information:

•    Whether technology-assisted review and/or keyword search methodologies are to be used to locate potentially relevant ESI.

•    Whether Victory may release the preservation hold and discard the non-relevant documents three months after its coding had been validated.

•    Whether 75% recall with no precision target is appropriate.

•    Are the validation parameters (95% Confidence Level, /-5% Confidence Interval, and 385 responsive document validation set) appropriate or should the sample size be increased?

•    Is the validation protocol proposed by Victory appropriate where it uses random samples of 385 documents from the set of documents predicted as responsive by the technology-assisted review software and random samples of 385 documents predicted as not responsive, and then comparing the two results?

•    Is Sam King obligated to undertake validation if its proposed keyword search protocol is permitted?

•    Must the validation review be blind without reference to the technology-assisted review or keyword search results? In other words, should the coders who are reviewing the validation set have no idea if the documents come from the responsive or non-responsive predicted categories of documents in order to remove any potential bias of knowing how the software actually coded the documents?

•    Is it appropriate to require parties to share some sampling of non-responsive documents to validate that their search approach worked?

Framing the ESI issues for the court in advance of the initial conference is critical. It permits the court to understand the complexities of the dispute up front, and such advance notice affords the court the opportunity to be able to craft procedures to achieve an orderly resolution of disputes.


Source(s) of Electronically Stored Information

The parties anticipate that discovery may occur from one or more of the following potential source(s) of electronically stored information.

Both sides have similar data repositories and anticipate they can agree on the form of production. If any party concludes that any of the sources of information listed above are inaccessible or that collection from or the search of any of those sources would be unduly burdensome, the parties will meet and confer in an attempt to resolve the matter. Parties will use their best efforts to raise any such objections as soon as possible.

Limitations on Production

The parties have discussed factors relating to the scope of production, including but not limited to: (1) number of custodians; (2) identity of custodians; (3) date ranges for which potentially relevant data will be drawn; (4) locations of data; (5) timing of productions (including phased discovery or rolling productions); and (6) electronically stored information in the custody or control of non-parties. To the extent the parties have reached agreements related to any of these factors, describe below:

Custodians: Both parties will exchange custodian lists as described in section ____ of the Joint Report on ________ 2014. Each party will have 21 days to file an objection and the parties will meet and confer to resolve any disputes.

Date Range: The default date range of discoverable documents and data is from _________ [which is when the initial design concept was conceived] through __________ [the date the lawsuit was commenced]. However, the parties agree that any party may propose a different date range for any particular custodian or type of data or documents.

Locations of Data: As noted above, the parties intend to hold a series of meet and confer sessions to determine the appropriate limits of ESI collection and production, finalize each party’s plan, and develop a schedule for the rolling production of documents intended to facilitate an orderly and manageable production consistent with the proposed case schedule.

If there is a disagreement regarding custodians, date ranges and locations, those areas need to be addressed early because they may each have a significant impact on the expense associated with collection, preservation and production of ESI.

Form(s) of Production

•    The parties have reached the following agreements regarding the form(s) of production:

Production issues should be detailed. However, production specifications are beyond the scope this article.

The parties have a working draft of the specifications for production of ESI and hard copy documents. During the upcoming negotiations concerning document collection and production, the parties will work toward finalizing these specifications and alert the court to any disputes arising therefrom. Unless otherwise specified below, the production will be in TIFF format with a Relativity load file.

•    Please specify any exceptions to the form(s) of production indicated above:

     The parties will initially produce spreadsheets in a form that provides for them to be readable and, on a case-by-case basis, will decide whether specific documents will be produced in native format.

•    The parties anticipate the need for judicial intervention regarding the following issues concerning the form(s) of production:

     There will need to be a determination of the scope of data that must be searched for ESI and whether the search process should utilize technology-assisted review or be keyword driven.

Privileged Material

•    Identification. The parties have agreed to the following method(s) for the identification (including the logging, if any; or alternatively, the disclosure of the number of documents withheld), and the redaction of privileged documents:

     The parties will follow the protocol described in the law review article by Judge John M. Facciola and Jonathan M. Redgrave “Asserting and Challenging Privilege Claims in Modern Litigation: The Facciola-Redgrave Framework,” 4 Fed. Courts L. Rev. 19 (2009).

•    Inadvertent Production/Claw-Back Agreements. Pursuant to Fed R. Civ. Proc. 26(b)(5) and Federal Rule of Evidence  502(e) (F.R.E.), the parties have agreed to the following concerning the inadvertent production of privileged documents:

     Given the size and timing of production, the parties will jointly seek the entry of an appropriate order, pursuant to F.R.E. 502(d), to address the attorney-client and attorney work product privileges regardless of inadvertence, and will include appropriate claw-back provisions in a proposed order.

•    The parties have discussed a 502(d) Order. (Yes)

     The provisions of any such proposed order shall be set forth in a separate document and presented to the court for its consideration.

Cost of Production

The parties have analyzed their clients’ respective data repositories and have estimated the costs associated with the production of electronically stored information. The factors and components underlying these costs are estimated as follows:

•    Costs – Plaintiff(s):

     Technology-assisted review involves sampling and language training of a software algorithm. Victory believes that such sampling will cost far less money and be far more effective than guessing at keywords and having to review all the false positive hits containing keywords. Keyword searching is more expensive than technology-assisted review coding but, more importantly, Victory submits that the keyword search protocol proposed by Sam King will not effectively identify sufficient relevant ESI. Many academic studies, as well as actual courtroom experience, indicate that keywords overlook a significant amount of relevant ESI.

     At this point, there is not a good estimate on the potential amount of relevant ESI. Thus, without more information, including keyword validation as discussed above, there is no basis to evaluate the effectiveness of the proposed keyword search protocol, and thus the cost of review using such technique. Sam King is relying on keywords to locate responsive documents and is not going to do any sampling on the documents which do not contain any of the keywords. Relying on technology-assisted review and sampling to validate results is a more effective and economical approach to identifying potentially relevant documents in this case.

Lastly, Sam King’s proposal to save money using offshore reviewers and contract lawyers will not result in huge cost savings compared to having technology perform this same task of culling non-responsive ESI from the review set.

•    Costs – Defendant(s):

     Sam King believes that costs can be controlled by using keywords, appropriate date ranges, and agreed upon custodians to derive a manageable data set. Costs can be further controlled by employing offshore reviewers (costing $35 an hour), supervised by counsel, to reduce the data set to a final review set, as opposed to utilizing law firm associates, whose billing rates exceed $400 an hour. Once the review set is generated by the contract review team, counsel will do a second-pass review to determine if such documents are responsive. If statistical sampling is required, Sam King submits that a larger sample size than that proposed by Victory in connection with its technology-assisted review solution is necessary in order to provide sufficient comfort in the results of its review. Sam King is uncomfortable using a smaller sample. It would propose instead a sample size calculated based on a 99% Confidence Level with a Confidence Interval of /- 2%.

This is one of the most useful sections of the Joint Order. It forces counsel to attempt to quantify how much their proposed collection and search methodologies will cost. When the court looks at the beginning of the Joint Order for the amount in controversy and then reviews the cost estimates of each of the parties, it is presumptively easier for the court to address the expense of e-discovery when it is able is review early on the parties’ respective views on proportionality and reasonableness.

•    Cost Allocation. The parties have considered cost-shifting or cost sharing and have reached the following agreements, if any:

     The parties agree to bear their own costs of discovery, without prejudice to any future application seeking cost-shifting.

•    Cost Savings. The parties have considered cost-saving measures, such as the use of a common e-discovery vendor or a shared document repository, and have reached the following agreements, if any: (None.)

Judicial Intervention

•    The parties anticipate the need for judicial intervention regarding the following issues concerning the production of electronically stored information: (To be determined.)

 The preceding constitutes the agreement(s) reached, and disputes existing, (if any) between the parties to certain matters concerning e-discovery as of this date. To the extent additional agreements are reached, modifications are necessary, or disputes are identified, they will be outlined in subsequent submissions or agreements and promptly presented to the court.


Once the Joint Order is filed with the court, the court would work with the parties to resolve the disputed issues.  There is no reason why the Joint Order could not be supplemented as the parties proceed through the steps of a complex e-discovery action and as e-discovery rulings are made.  The Joint Order attempts to keep the parties focused on where there are disagreements and it provides a dispute resolution framework to address issues as they arise.  This is especially true where a party is seeking to utilize technological advances in the face of opposition.

One concept that comes clear from this fact pattern is that Victory is attempting to save money with their proposed protocol and embraces more transparency than many parties are accustomed in an attempt to provide comfort to the opposing party that the proposed process will achieve its objective.  Sam King, on the other hand, is willing to spend more money than Victory, but is opting for the traditional approach of no transparency as it seeks to have every document personally reviewed.  This creates a real challenge to practitioners, clients and the courts.  The Joint Order is a useful tool to frame issues so that a court can see the pros and cons of different technological approaches suggested by parties.

Karl Schieneman ( the principal author of this article, is President of Review Less, a national consulting and document review company which specializes in designing predictive coding review workflows and creator of the National Predictive Coding Thought Leadership Series CLE program.

Mark A. Berman ( a partner at Ganfer Shore Leeds & Zauderer, LLP and a member of the Electronic Discovery Committee and Co-Chair of the Social Media Committee of the Commercial and Federal Litigation Section of the New York State Bar Association, assisted in the preparation of this article.

The foreword is authored by United States Magistrate Judge James C. Francis IV from the Southern District of New York.

[2] At LegalTech 2013, a panel consisting of a judge, practitioners, and vendors decided to enact a mock conference highlighting a technology-assisted review dispute and how much the Joint Order would make an initial court conference more targeted on the specific e-discovery issues and disputes.  Magistrate Judge James C. Francis and Karl Schieneman were participants in this mock conference, as well as Herbert Roitblat, Conor P. Crowley, Ariana J. Tadler and Thomas C. Gricks, III.  This article takes the mock conference further and seeks to makes the scenario more applicable to general commercial litigation.

[3] The positions taken by the parties in this hypothetical are not intended to be considered “best practices” in e-discovery.  The purpose of this article is to highlight how disputes in complex litigation can be clarified and “joined” for resolution by the court through the Joint Order.  The commentary of how the Joint Order seeks to facilitate cooperation and to identify undisputed or disputed issues is contained in italicized text.  This commentary is not part of the Joint Order and is drafted by the authors to provide explanatory guidance.

[4] Even though technology-assisted review is proposed by Victory, attorneys and technologists will differ as to how this or any protocol should be implemented, and some of the positions taken below by Victory may be viewed as controversial.  As such, while the below may serve as a guideline, any technology-assisted review protocol must be tailored to the facts of the case and the legal and strategic significance of each step must be well-thought out and clearly analyzed.

[5] The 385 document total used in this fact pattern is the total number of documents from a random sample utilized in most collections noted in electronic discovery cases needed to achieve a “Confidence Level” of 95% and a “Confidence Interval” of /- 5. An approximation of this total can be found by using online sample calculators such as those provided by or

[6] The Confidence Level and Confidence Interval are selected by a party to determine the size of the sample to be used.  The sample size in this example is 385 given the number of documents in the population and it can be calculated by using a formula once the actual population of documents to be reviewed is determined and the desired Confidence Level and Confidence Interval are chosen.

 A confidence interval expresses the degree of uncertainty associated with a sample estimate.  It is a combination of a range, combined with a probability statement.  For example, we may say that an election poll has candidate A leading with 27% of the vote relative to candidate B with 23%.  Both percentages estimate the proportion of the voters who are likely to vote for the two candidates and are accurate within /-5.35%.  The /-5.35% is the confidence interval at the 95% confidence level.  Roughly, this means that if the election were held today, predictions based on this poll would be with /-5.35% of the true vote 95% of the time.  The larger the sample size, all other things equal, the smaller the confidence interval.  The poll mentioned here interviewed 336 likely voters.  A sample of 1,000 likely voters would have a 95% confidence interval of /-3.10%.

 . . . A confidence level indicates the degree of confidence one has in the estimate derived from a sample.  Roughly, it is the likelihood that the true value from the population lies within the range specified by the confidence interval.  If you did [sic] 100 experiments with a 95% confidence level, then 95 of those experiments would find the true value being estimated is included within the appropriate confidence interval for that experiment.

 Herbert L. Roitblat, Statistics and Sampling for eDiscovery: Glossary and FAQ, OrcaTec LLC, 2011 (emphasis in the original).

 In summary, the concepts of Confidence Level (95%) and Confidence Interval ( /- 5%) are statistical concepts which work in tandem with each other based on the tradeoff of the amount of risk the sampling party is willing to bear in not identifying the relevant information and the level of effort they want to undertake to measure the results. For instance, it takes more resources to review a sample of 2,400 documents or more when compared to 385 documents when the degree of confidence provided by a smaller sample size is adequate for the needs of the particular case.

 [7] Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 189–90 (S.D.N.Y. 2012).

The objective of review in ediscovery is to identify as many relevant documents as possible, while reviewing as few non-relevant documents as possible. Recall is number of responsive documents found over the estimated total number in a collection; precision is the fraction of identified documents that are relevant out of the total documents found. The goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the “value” of the case. See, e.g., Maura R. Grossman & Gordon V. Cormack, Technology–Assisted Review in E–Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Rich. J.L. & Tech., Spring 2011, at 8–9, available at http:// jolt.richmond. edu/vl 7 i 3/ article 11. pdf.