Fixing Logical Models API Errors with EBEAN_MAX_TRANSACTION_RETRY Configuration – DataHub Customer Support Portal

Area: API Issues

Sub-Area: Logical Models API / Transaction Collision Errors

Issue

When using the Logical Models API to link schema fields of a dataset to its parent columns — for example, via linkPhysicalChild, unlinkPhysicalChild, or setLogicalParent mutations — GMS may return transaction-related errors during parallel or high-concurrency write operations. This is most commonly observed when a backfill job issues many link calls simultaneously, causing row-level lock collisions in the underlying database. The errors can manifest as transient failures that succeed on retry but become persistent under sustained parallel load.

Error Messages

javax.persistence.RollbackException: Transaction rolled back
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
org.springframework.dao.CannotAcquireLockException: could not execute statement

You Might Be Asking

Why do Logical Models API calls fail intermittently when run in parallel?
How do I stop deadlock errors when backfilling logical model relationships at scale?
What does EBEAN_MAX_TRANSACTION_RETRY do and when should I increase it?
Is this a bug in DataHub, or is there a configuration change I can make to resolve it?

Solution

Understand the root cause.
When multiple concurrent API calls write to the logicalParent aspect — for instance, from a parallelized backfill script — they can compete for the same database row locks. Ebean ORM, which DataHub uses internally, will retry failed transactions automatically up to a configured maximum. If the default retry count is too low, collisions that would otherwise self-resolve are surfaced as errors to the caller.
Apply the short-term mitigation: increase EBEAN_MAX_TRANSACTION_RETRY.
Set the environment variable EBEAN_MAX_TRANSACTION_RETRY to 7 (or higher) on both the GMS pod and the MCE consumer pod. This widens the retry window so that transient lock collisions resolve on their own before the error is surfaced.

For Docker Compose deployments, add the variable to the relevant service definitions in your docker-compose.yml:
```
services:
  datahub-gms:
    environment:
      EBEAN_MAX_TRANSACTION_RETRY: "7"

  datahub-mae-consumer:
    environment:
      EBEAN_MAX_TRANSACTION_RETRY: "7"
```
For Kubernetes deployments, patch the relevant Deployments:
```
kubectl set env deployment/datahub-gms \
  EBEAN_MAX_TRANSACTION_RETRY=7 \
  -n <your-namespace>

kubectl set env deployment/datahub-mae-consumer \
  EBEAN_MAX_TRANSACTION_RETRY=7 \
  -n <your-namespace>
```
Or add it directly to your Helm values.yaml:
```
datahub-gms:
  extraEnvs:
    - name: EBEAN_MAX_TRANSACTION_RETRY
      value: "7"

datahub-mae-consumer:
  extraEnvs:
    - name: EBEAN_MAX_TRANSACTION_RETRY
      value: "7"
```
Restart the affected pods after applying the change.

Serialize parallel backfill calls where possible.

In addition to the retry increase, reducing the concurrency of your backfill job significantly lowers the likelihood of collisions. If your job issues link calls in parallel, consider adding a throttle or batching requests sequentially:

# Example: throttle Python backfill script to 1 request at a time
import time

for field_mapping in field_mappings:
    response = link_schema_field(field_mapping)
    if not response.ok:
        print(f"Failed: {response.text}")
    time.sleep(0.1)  # Add delay to reduce lock contention

Verify the fix.
After applying EBEAN_MAX_TRANSACTION_RETRY=7 and restarting the pods, re-run the backfill job (or a representative subset of it) and check the GMS logs for any remaining transaction rollback errors:
```
kubectl logs -n <your-namespace> \
  deployment/datahub-gms \
  --since=30m \
  | grep -i "rollback\|deadlock\|lock"
```
If no deadlock or rollback errors appear, the mitigation is effective.
Plan for a durable fix.
The EBEAN_MAX_TRANSACTION_RETRY increase is a mitigation, not a permanent cure. A more durable fix involves a write mode that eliminates the row-level locks causing these collisions entirely. This change is tracked in DataHub's engineering backlog. Coordinate with your DataHub account team to include this fix in your next scheduled upgrade.

Additional Notes

This issue is most commonly triggered by parallelized backfill jobs that issue many linkPhysicalChild or setLogicalParent API calls simultaneously. Single-threaded or low-concurrency usage is unlikely to reproduce the error. The recommended value of EBEAN_MAX_TRANSACTION_RETRY=7 has been validated as an effective short-term mitigation; values higher than 10 are generally not necessary and may mask other underlying problems. Note also that a separate authorization enforcement change (introduced in recent versions) requires that the calling actor have EDIT_ENTITY privilege on both the child dataset and the parent logical model. If your GMS errors include UnauthorizedException or HTTP 403 responses rather than transaction rollback messages, refer to DataHub's policy documentation to grant the appropriate privileges to the calling service account. This article specifically addresses the transaction collision / deadlock variant of Logical Models API errors.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation