Skip to main content

Merge Duplicates: behavior, errors, and troubleshooting

The Merge Duplicates Flow Step consolidates records that share the same values in the selected Fields into a single row. In version 2025.2.4, Merge Duplicates changed and may produce different results than in earlier versions. This article explains what to expect in various scenarios. For general information, see Transform Flow Steps.

Processing scenarios​

Successful processing​

What you will see:

  • Your data flows through normally, sorted by your selected Fields.
  • Duplicate detection works and records are merged on your selected Fields.
  • Data in the remaining Fields is processed according to the Flow Step configuration.

No sortable data found​

What happened: none of your selected Fields contain values that can be sorted (all empty, null, or invalid) or merged upon.

What you will see:

  • All data records are merged into a single row.
  • An error is included in the data row.
  • Data is processed in the order it was received.
  • All records are still included in the single-row output.

What you can do:

  • Review your Flow Step configuration. Clear the selected Fields with the X and confirm you chose the correct ones.
  • If your data may contain blanks or invalid values, add a Flow Step (such as a Power Script) before Merge Duplicates to sanitize the data: validate Fields, apply default values, or ensure formatting is correct. This ensures your selected Fields always contain valid data for sorting and merging.

Some Fields cannot be sorted​

What happened: some of the selected Fields do not contain valid sortable values.

What you will see:

  • Certain selected Fields have no values throughout your results.
  • Processing continues, but only Fields with valid values are used for merging.
  • Fields without valid data are silently skipped (no warnings are shown unless all Fields are invalid or cannot be sorted).
  • All records are still processed and sorted by the remaining valid Fields.

What you can do:

  • If the empty Fields are unexpected, check their Datasources or preceding Flow Steps for issues. Some rows may also include error messages unrelated to Merge Duplicates.
  • If needed, add a Flow Step (such as a Power Script) before Merge Duplicates to validate data or assign default values so the Fields are sortable.
  • If the empty Fields are expected, no action is required.

Error scenarios and recovery​

Staging data processing errors​

What happened: an error occurred while staging your data in the temporary Elasticsearch index used for sorting before merging.

Possible causes:

  • Issues with your Elasticsearch cluster (connectivity, resource limits, or other failures).
  • Invalid data in at least one row for at least one selected Field, causing Elasticsearch to reject and fail to index it.

What you will see:

  • A single error record: the first record from your original data with error details attached.
  • Processing stops immediately to prevent data corruption.
  • Only the first record (with error details) is returned; all other upstream data is discarded.

What you can do:

  • Check connectivity to your Elasticsearch cluster.
  • Verify the cluster has sufficient resources and is healthy.
  • Add a Flow Step (such as a Power Script) before Merge Duplicates to validate or sanitize data and assign default values.

Sorting data processing errors​

What happened: an error occurred while retrieving the sorted, staged data from the temporary index.

Possible causes:

  • Issues with your Elasticsearch cluster (connectivity, resources, or other failures).
  • Data-specific exceptions when searching or sorting, for example if a selected Field contains unexpected values, types, or structures. This often happens when a Field includes complex objects or binary data instead of simple values.

What you will see:

  • Incomplete data output.
  • Partially processed results containing only the records processed before the error.
  • An error record as the final row, including details about the issue.
  • Processing stops to prevent data corruption.

What you can do:

  • Check connectivity to your Elasticsearch cluster.
  • Verify the cluster has sufficient resources and is healthy.
  • Add a Flow Step (such as a Power Script) before Merge Duplicates to validate or sanitize data and assign default values.
Also available through the REST API
Everything on this page is built on Informer's public REST API, the same one the product uses, so your scripts and integrations can do it too. Developer reference: Queries.