> You are reading the Docusign Developer Documentation. For clean Markdown of any page, append .md to the page URL.
>
> Documentation indexes by section:
> - APIs and SDKs: https://docusign.docs.buildwithfern.com/docs/llms.txt
> - Platform 101: https://docusign.docs.buildwithfern.com/platform/llms.txt
> - Extension Apps: https://docusign.docs.buildwithfern.com/extension-apps/llms.txt
> - Docusign CLI: https://docusign.docs.buildwithfern.com/docusign-cli/llms.txt
>
> Authentication (OAuth, JWT Grant, M2M): https://docusign.docs.buildwithfern.com/platform/auth
> Webhooks and Connect: https://docusign.docs.buildwithfern.com/platform/webhooks
>
> OpenAPI specifications: https://docusign.docs.buildwithfern.com/openapi.json
> MCP server: https://docusign.docs.buildwithfern.com/_mcp/server

# Test agreement customizations

> Learn how to test AI extraction accuracy for your Agreement Manager customizations using the Docusign CLI. Generate a testing template, enter ground truth values, and run the test.

Testing enables you to measure the accuracy of Agreement Manager's AI extractions before deploying your configuration to production. By comparing AI-extracted values against known correct values, you can identify fields or agreement types that need improvement and iterate on your configuration until the extraction accuracy meets your requirements.

The testing process has four steps:

1. [Set up test agreements](/docusign-cli/test-agreement-customizations#step-1-set-up-test-agreements)
2. [Generate the testing template file](/docusign-cli/test-agreement-customizations#step-2-generate-the-testing-template-file)
3. [Update the testing template file with ground truth values](/docusign-cli/test-agreement-customizations#step-3-update-the-testing-template-file-with-ground-truth-values)
4. [Run the test](/docusign-cli/test-agreement-customizations#step-4-run-the-test) to compare AI-extracted values against those ground truth values

For a walkthrough of the testing process, see this video:

## Prerequisites and test agreement requirements

Before you begin testing, make sure that:

* You are authenticated to the target environment. See [Authenticate](/docusign-cli/authenticate) for details.
* You have [uploaded your Agreement Manager package](/docusign-cli/configure-agreement-types-and-fields/upload-package) to configure the agreement types and fields you want to test.

For the test agreements themselves, follow these requirements and recommendations:

* Add your test agreements to the following directory: `{workspaceName}/{projectName}/agreement-manager/files/test/`
* Test agreements must be different from the [training agreements](/docusign-cli/configure-agreement-types-and-fields/create-package#training-documents) used to train AI extraction. They should represent realistic agreements expected in production.
* Use a maximum of 400 agreements per test run. This limit is based on guidance from the Docusign Applied Science team. A diverse set of 400 agreements produces accuracy measurements that are statistically equivalent to testing across hundreds of millions of agreements.
* Include a diverse mix of agreement types and field values to ensure accuracy measurements reflect real-world conditions.
* The following file formats are supported:

  * PDF
  * Word: DOC, DOCX, RTF
  * Excel: XLS, XLSX, XLSB
  * PowerPoint: PPT, PPTX, PPSX
  * HTML
  * Images: JPG, PNG, TIF, TIFF

### Sample agreements

Docusign provides [sample agreements](https://support.docusign.com/s/document-item?language=en_US\&bundleId=grn1724866350175\&topicId=amt1724866817262.html) that you can use as training or test documents while you are learning how the Docusign CLI works. Because these agreements are generic, you should not use them to prepare Agreement Manager customizations for the production environment. When working toward a production deployment, use training and test documents that are representative of your organization's or customer's agreements.

## Step 1. Set up test agreements

The first step in the testing process is uploading test agreements to Agreement Manager. After the upload, Docusign extracts agreement values for standard fields and AI-enabled custom fields. These extracted values will be included in the [testing template file](/docusign-cli/test-agreement-customizations#step-2-generate-the-testing-template-file) if you [prefill values](/docusign-cli/test-agreement-customizations#prefill-the-testing-template) when generating the template.

See [Prerequisites and test agreement requirements](/docusign-cli/test-agreement-customizations#prerequisites-and-test-agreement-requirements) for information about preparing test agreements for upload.

To upload test agreements, run this command:

`$ ds agm test --setup`

When you run the `test --setup` command, the CLI checks the `{workspaceName}/{projectName}/agreement-manager/files/test/` directory and detects whether any of the files have been uploaded previously. The CLI then displays these prompts:

* `Found N test agreements that have not been ingested into Agreement Manager yet. Do you want to ingest N test agreements into Agreement Manager?`
  Respond with `y` to upload agreements that have not been uploaded before.

  **Note:** When the CLI generates the testing template file, it includes all files in the `test` directory, regardless of whether they have been uploaded. If you try to run a test when the testing template file contains files that have not been uploaded, the `test --run` command will return an error.
* `Found N agreements in Agreement Manager that need AI reprocessing. Do you want to reprocess N agreements to ensure AI extracted values are accurate?`
  Respond with `y` to reprocess previously uploaded test agreements. Docusign recommends reprocessing test agreements if you've made any of the following updates to `agreement-manager-manifest.json` since the agreements were uploaded:

  * Added new custom fields
  * Added standard or custom fields to agreement types
  * Made changes to [improve AI extraction accuracy](/docusign-cli/test-agreement-customizations#improve-ai-extraction-accuracy)

## Step 2. Generate the testing template file

The testing template file, `testing.csv`, defines the structure of your test run. It contains one row per test agreement and one column for each standard and custom field. The contents of the file are based on the test agreements you uploaded using the `ds agm test --setup` command, described in [Step 1. Set up test agreements](/docusign-cli/test-agreement-customizations#step-1-set-up-test-agreements).

You populate `testing.csv` with the expected (ground truth) values that the AI should extract, and the CLI uses it as the basis for comparison when you run the test.

To generate `testing.csv`, run the following command:

`$ ds agm test --generate-test-template`

#### Additional command options

When you run `test --generate-test-template`, you can include these additional options:

| Option                  | Description                                                                                                                                                                                                               | Default value                                                       |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `--bypass`              | Skips confirmation prompts.                                                                                                                                                                                               | `false`                                                             |
| `-y`                    | Alias for `--bypass`. Skips confirmation prompts.                                                                                                                                                                         | `false`                                                             |
| `--prefill-extractions` | Prefills the CSV with the AI-extracted agreement type and field values for each test agreement. See [Prefill the testing template](/docusign-cli/test-agreement-customizations#prefill-the-testing-template) for details. | `false`                                                             |
| `--output <path>`       | Sets the output path (including filename) where the testing template is written.                                                                                                                                          | `{workspaceName}/{projectName}/agreement-manager/tests/testing.csv` |

#### What happens when the command runs

When you run `test --generate-test-template`, the CLI generates the `testing.csv` file to the default location `{workspaceName}/{projectName}/agreement-manager/tests/` or to the location specified in the `--output` option.

If a `testing.csv` file exists in the output location, the CLI prompts you to confirm whether the existing file should be overwritten. Save a backup of any manually entered ground truth values before overwriting the file.

> **Note**
>
> If the package was recently [uploaded](/docusign-cli/configure-agreement-types-and-fields/upload-package), the AI may not have finished extracting values from the training documents yet. If the `testing.csv` file contents are not as expected, wait a few minutes and rerun the command before proceeding.

### Prefill the testing template

The `--prefill-extractions` option instructs the CLI to populate `testing.csv` with the actual AI-extracted values for all relevant standard and custom fields, rather than leaving the cells blank. This gives you a starting point and can help you identify which values the AI has already extracted correctly.

Example command:

`$ ds agm test --generate-test-template --prefill-extractions`

This option offers a useful advantage: instead of entering every ground truth value from scratch, you can review the prefilled values, correct any that are wrong, and leave accurate ones unchanged. This can significantly reduce the time required to prepare the testing file.

Use this option with care. If you run the test without reviewing and correcting the prefilled values, the test results will always show 100 percent accuracy for the prefilled cells because the ground truth and the extracted values will be identical. Always review every prefilled cell and update any values that do not match the actual content of the agreement.

> **Note**
>
> If the package was recently [uploaded](/docusign-cli/configure-agreement-types-and-fields/upload-package), the AI may not have finished extraction on the training documents yet, and some cells may appear blank even when using `--prefill-extractions`. If this happens, wait a few minutes and rerun the command to allow the AI extractions to complete. You can also populate the values manually if needed.

## Step 3. Update the testing template file with ground truth values

Open `testing.csv` and enter the expected values for the relevant fields in each test agreement. The values you enter are treated as the ground truth and compared against the AI-extracted values during the test run.

Each row in `testing.csv` corresponds to one test agreement and includes the agreement name, the agreement type, and columns for all standard and custom fields. You do not need to populate every field for every agreement. Leave a cell blank if the field is not present in that agreement or is not relevant for testing. Blank cells are ignored during the test run.

Keep the following formatting rules in mind when entering values:

* For standard duration fields (such as *Liability Cap Duration*, *Renewal Notice Period*, and *Renewal Term*), enter values in the format `# {days/months/years}`. Examples: 30 days, 6 months, 3 years.
* DateTime values can be entered in ISO format or `MM/DD/YYYY` format. For example, if an agreement has an expected start date of 03/28/2026, enter that value in the Start Date column for that row.

## Step 4. Run the test

After you have populated `testing.csv` with ground truth values, run the test with the following command:

`$ ds agm test --run`

#### Additional command options

When you run `test --run`, you can include this additional option:

| Option            | Description                                                                                                 | Default value |
| ----------------- | ----------------------------------------------------------------------------------------------------------- | ------------- |
| `--partial-match` | If `true`, string values are compared by token (word). If false, string values are compared by exact match. | `false`       |

#### What happens when the command runs

When you run `ds agm test --run`, the CLI fetches the AI-extracted values from your test agreements in Agreement Manager, compares them against the ground truth values in `testing.csv`, and calculates accuracy results. Upon completion, the CLI outputs a summary of field-level accuracy:

![Test results](/_fern-files/_img/064d4438de3a13da1dbcde2c3057e511545117f6a3d7af7b2e8e73beaf7c14e4.webp)

The CLI also generates two result files in `{workspaceName}/{projectName}/agreement-manager/tests/`:

* `{current date-time}-field-test-results.csv`: Contains a row for every tested field. For each test agreement, this file shows whether the AI-extracted value matched the ground truth value in `testing.csv`.
* `{current date-time}-agreement-type-test-results.csv`: Contains a row for every test agreement. For each agreement, this file shows whether the AI-extracted agreement type matched the agreement type in `testing.csv`.

> **Note**
>
> If the package was recently [uploaded](/docusign-cli/configure-agreement-types-and-fields/upload-package), the AI may not have finished extraction on the training documents yet. If extracted values appear missing or incomplete in the results, wait a few minutes and rerun the command.

## Improve AI extraction accuracy

If the accuracy results from your test run do not meet your requirements, you can improve them by refining the configuration in your `agreement-manager-manifest.json` file and retraining the AI. The available improvements depend on the type of extraction.

**For custom fields:**

* Improve the field's `aiDefinition` so the AI has a clearer understanding of what the field represents.
* Add more training examples to the `examples` section (3–5 examples are recommended).
* Use full paragraphs in the `exampleText` property to provide richer context for the AI.
* Refine existing examples that may be ambiguous or unrepresentative.

**For custom agreement types:**

* Improve the agreement type's `aiDefinition` to give the AI a clearer understanding of the type.
* Add more diverse training agreements (5–10 documents are recommended).
* Ensure training agreements represent real-world variations of the agreements you expect in production.

**Note:** Improving the accuracy of standard extractions is not supported.

After making changes to the package to improve accuracy, follow these steps to check the results:

1. [Upload](/docusign-cli/configure-agreement-types-and-fields/upload-package) the updated package. Changes to the manifest do not take effect until the package is uploaded.
2. Complete the [test agreement setup](/docusign-cli/test-agreement-customizations#step-1-set-up-test-agreements) to apply the updated configuration and reprocess test files.
3. If you've added test agreements, update the [testing template file](/docusign-cli/test-agreement-customizations#step-2-generate-the-testing-template-file).
4. [Rerun the test](/docusign-cli/test-agreement-customizations#step-4-run-the-test).

## Frequently asked questions

### How many test agreements should I use?

You can use up to 400 agreements per test run. The Applied Science team has determined that a diverse set of 400 agreements produces accuracy results that are statistically equivalent to testing across hundreds of millions of agreements. Prioritize diversity of agreement types and field values over quantity.

### What does it mean if a cell in testing.csv is blank after using --prefill-extractions?

A blank cell after using `--prefill-extractions` typically means the AI has not yet finished processing that agreement or field. This can happen when agreements were recently uploaded. Wait a few minutes and rerun the `test --generate-test-template` command to allow extraction to complete.

### Can I reuse testing.csv across multiple test runs?

Yes. If you rerun `test --generate-test-template` and a `testing.csv` file exists in the output location, the CLI prompts you to confirm whether the existing file should be overwritten. Save a backup of any manually entered ground truth values before overwriting the file.

### Why does my test run show 100 percent accuracy?

If you used the `--prefill-extractions` option and did not review and update the prefilled values before running the test, the ground truth and extracted values will be identical, resulting in 100 percent accuracy. Always review every prefilled cell and correct values that do not match the actual agreement content.

### Can I improve the accuracy of standard field extractions?

No. Improving the accuracy of standard field extractions is not supported. You can only improve accuracy for custom fields and custom agreement types where `customExtractionEnabled` is set to `true`.

## Next steps

* Once accuracy meets your requirements, [deploy your agreement customizations](/docusign-cli/deploy-agreement-customizations) to a production account.
* Learn about [bulk ingesting agreements in production](/docusign-cli/bulk-ingest-agreements), including supported file types and how duplicate detection works.
* If you deploy customizations to an environment where agreements have already been ingested, initiate a [reprocess](/docusign-cli/reprocess-agreements) to apply the customizations to existing agreements.