Evaluating an AI tool for a law firm: the confidentiality questions to answer before a pilot

ABA Model Rule 1.6 provides in paragraph (c) that a lawyer shall make reasonable efforts to prevent the inadvertent or unauthorized disclosure of, or unauthorized access to, information relating to the representation of a client. A new AI tool is one more place that information could go, so it falls squarely inside that duty.

ABA Formal Opinion 512, issued July 29, 2024, addresses generative AI specifically. It discusses self-learning tools into which lawyers input information relating to a representation and concludes that, because many such tools are designed so that their output could lead to disclosure of that information, the client's informed consent is required before inputting it. It also states that general, boilerplate provisions in engagement letters purporting to authorize AI use are not sufficient for informed consent. Your jurisdiction may have its own opinions, and they may differ.

The first questions are about data handling. Does the vendor use submitted content to train or improve models, for this customer or for others? Can that be turned off by contract rather than by a setting a user could change? How long are prompts, documents and outputs retained, and does retention continue after the subscription ends?

Opinion 512 lists similar considerations, including investigating the tool's reliability, security measures and policies, and determining whether the tool retains information submitted by the lawyer before and after discontinuation of services or asserts proprietary rights to it. I would want the answers in the contract or a data processing agreement, not only in a marketing page.

Next, access. Which vendor personnel can see customer content, under what circumstances, and is that access logged? Which subprocessors handle the data, and where? Can the firm restrict access within its own tenant by practice group, matter or ethical wall?

Ethical walls deserve a specific question. If the tool includes search or retrieval across the firm's documents, confirm that it enforces the same matter-level access restrictions as the document management system. A tool that lets a user ask questions across all firm documents can bypass walls without anyone intending it.

Confidentiality is not the only issue. Ask how the tool cites its sources, whether citations link to the actual source documents, and what the vendor says about accuracy limitations. The firm will still need lawyers to verify every output used in client work, and the pilot should test how practical that verification is.

Once the answers are in, decide what data the pilot may use. Options range from public information only, to internal non-client material, to client information for matters where the client has given informed consent. Write the rule down and make it visible to everyone in the pilot.

I would start with the narrowest category that still tests the tool meaningfully. Expanding to client information later, with the right consents and contract terms in place, is easier than unwinding a pilot that started too broadly.

Many tools have settings for retention, training opt-out, sharing and access. Configure them before users are invited, record the configuration, and check it again after vendor updates. Restrict the pilot to named users who have been told the data rules.

If the tool integrates with email or document systems, review exactly what permissions the integration requests. A drafting tool rarely needs permission to read every mailbox in the firm.

Ask how to export the firm's data and how to confirm deletion if the pilot ends. Record the vendor's commitments on deletion timing and backups. A pilot that ends with no clear deletion confirmation leaves an open question about where client information sits.

Store the questions, the vendor's answers, the contract terms, the configuration and the pilot's data rules together. If a client or regulator later asks how the firm evaluated the tool, that record is the answer. It also makes the next tool evaluation faster.

The acceptance condition I would set before client information enters the pilot: written answers on training, retention, access and subprocessors, contract terms that match those answers, ethical walls tested, the configuration recorded, informed consent obtained where the firm's analysis requires it, and named pilot users who know the data rules.

Bring the tool you are evaluating, its contract and settings, and the matters you would like to pilot.