Databricks Spark Lineage: Setup Guide for acryl-spark-lineage on DBR 17.3 LTS with Unity Catalog

Area: Ingestion Issues

Sub-Area: Spark Lineage / Databricks Runtime Configuration

Issue

Setting up acryl-spark-lineage on Databricks Runtime 17.3 LTS (Scala 2.13, Spark 4.0, Java 17) involves several interrelated configuration steps that must all be correct before Spark lineage flows into DataHub. Common failure modes include a driver crash caused by an OpenLineage BigQuery class-probe bug in older agent versions, a ClassNotFoundException when the agent JAR is not on the system classpath at driver startup, a Scala version mismatch between the installed JAR and the runtime, auth failures caused by using a personal access token instead of a service account token, and missing table links in DataHub when Unity Catalog names are not propagated to the agent. This article consolidates the full working setup and explains each pitfall.

Error Messages

• java.lang.NoClassDefFoundError: scala/Serializable at io.openlineage.spark.agent.util.BigQueryUtils.hasBigQueryClasses
• java.lang.ClassNotFoundException: datahub.spark.DatahubSparkListener not found in com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader
• java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)'
• java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.scheduler.SparkListenerJobStart.stageIds()'
• SparkMisconfiguration: Spark Driver was down due to misconfiguration, please check your config.
• User urn:li:corpuser:<username> is unauthorized to modify entity (HTTP 403 from GMS)
• AssertionError: SharedState should not have already initialized
• ERROR DriverDaemon$: Fatal uncaught exception. Terminating driver.

You Might Be Asking

• Why does my Spark driver crash immediately after DatahubSparkListener registers successfully?
• Why does datahub.spark.DatahubSparkListener throw a ClassNotFoundException even though the JAR is attached as a cluster library?
• Why does my init script install the _2.12 JAR on a Scala 2.13 runtime?
• Why does DataHub show the Spark job but no upstream or downstream tables?
• Which version of acryl-spark-lineage supports DBR 17.3 LTS and Spark 4.0?
• Does acryl-spark-lineage support Spark 4.2 / DBR 19.x?

Solution

Step 1 — Use the correct agent version and JAR artifact

Older releases of acryl-spark-lineage bundle an OpenLineage version whose BigQueryUtils.hasBigQueryClasses probe catches only Exception. On Scala 2.13 runtimes, the missing scala.Serializable class surfaces as a NoClassDefFoundError (an Error, not an Exception), which escapes the probe and crashes the SparkContext. This was fixed in OpenLineage 1.50.0, which is bundled in acryl-spark-lineage version 1.7.0.14.

• Use artifact: io.acryl:acryl-spark-lineage_2.13:1.7.0.14
• Do not use 1.7.0 (contains 669 duplicate class entries that cause crashes on Parquet scans).
• Do not resolve "latest" from Maven; the <release> pointer may point to a release candidate.
• DBR 17.3 LTS is currently the highest supported runtime. Spark 4.2 (DBR 19.x) is not yet supported — the agent calls CatalogManager as a class, which became an interface in Spark 4.2, causing an IncompatibleClassChangeError on any catalog operation.

Step 2 — Stage the JAR via an init script, not the Libraries UI

spark.extraListeners classes are instantiated during SparkContext initialization, which runs before Databricks attaches cluster libraries or spark.jars.packages artifacts. The agent JAR must be present in /databricks/jars before the driver starts. Placing it there via a cluster init script is the only supported method.

1. Upload the JAR to a Databricks volume or DBFS path accessible from the cluster:

   databricks fs cp --overwrite acryl-spark-lineage_2.13-1.7.0.14.jar dbfs:/datahub/

2. Create a cluster init script with the following content and attach it to the cluster. Use this script only on DBR 17.3 LTS clusters (17.3.x-scala2.13):

   #!/bin/bash
   set -euo pipefail
   
   JAR_NAME="acryl-spark-lineage_2.13-1.7.0.14.jar"
   curl -sSL --fail \
     -o "/databricks/jars/${JAR_NAME}" \
     "https://repo1.maven.org/maven2/io/acryl/acryl-spark-lineage_2.13/1.7.0.14/${JAR_NAME}"
   
   echo "DataHub Spark lineage agent installed: $(ls -lh /databricks/jars/${JAR_NAME})"

3. After the cluster restarts, verify exactly one acryl JAR is present:

   %sh
   ls -la /databricks/jars | grep -iE 'acryl|scala-library'

   Expected output: one line for acryl-spark-lineage_2.13-1.7.0.14.jar (~99 MB). If a _2.12 JAR is also present, a conflicting init script is installing it — remove all duplicate init scripts before retrying.

Step 3 — Detect Scala version correctly in a dynamic init script

If you need a single init script that works across Scala 2.12 and Scala 2.13 runtimes, detect the Scala version by probing the JAR files Databricks places in /databricks/jars. Note that the filename pattern for the Scala standard library JAR varies between DBR versions — use scala-library as the probe target rather than a versioned glob, and validate the selection after installation.

#!/bin/bash
set -euo pipefail

VERSION="${DATAHUB_SPARK_LINEAGE_VERSION:-1.7.0.14}"

# Detect Scala version from the runtime JAR layout
SCALA_SUFFIX="2.12"
if ls /databricks/jars/scala-library-2.13*.jar 2>/dev/null | grep -q .; then
  SCALA_SUFFIX="2.13"
fi

JAR_NAME="acryl-spark-lineage_${SCALA_SUFFIX}-${VERSION}.jar"
MAVEN_URL="https://repo1.maven.org/maven2/io/acryl/acryl-spark-lineage_${SCALA_SUFFIX}/${VERSION}/${JAR_NAME}"

echo "Installing DataHub Spark lineage agent (${VERSION}, Scala ${SCALA_SUFFIX}) on $(hostname)..."
curl -sSL --fail -o "/databricks/jars/${JAR_NAME}" "${MAVEN_URL}"
echo "Installed: $(ls -lh /databricks/jars/${JAR_NAME})"

Important: Always verify the detected Scala suffix matches the cluster runtime after installation. A mismatch loads the wrong artifact, which registers the listener successfully but then kills the SparkContext on the first Spark job with a NoSuchMethodError.

Step 4 — Configure Spark properties

Set the following Spark configuration properties on the cluster (via the Cluster UI under Advanced Options > Spark, or in the cluster JSON):

spark.extraListeners=datahub.spark.DatahubSparkListener
spark.datahub.rest.server=https://<your-datahub-host>/gms
spark.datahub.rest.token={{secrets/<scope>/<datahub-token-key>}}

Step 5 — Use a service account token, not a personal access token

The agent writes pipeline, job, and lineage entities to DataHub GMS on every Spark application start and job completion. Personal access tokens tied to user accounts that lack a DataHub role will produce HTTP 403 errors for all writes. The job and runs will not appear in DataHub.

1. In DataHub, go to Settings > Users & Groups > Service Accounts and create a new service account.
2. Assign the service account the Editor role (minimum required to create dataFlow, dataJob, and dataset entities).
3. Generate a token for the service account.
4. Store the token in a Databricks secret scope:

   databricks secrets put --scope <scope-name> --key <datahub-token-key>

5. Reference the secret in the Spark config (as shown in Step 4). The cluster owner must have read access to the secret scope.
6. Revoke any personal token that was previously used in the Spark config. Do not store tokens in plain text in init scripts, notebooks, or attached files.

Step 6 — Align table names with Unity Catalog ingestion (platform alias)

By default, the agent records tables using their Hive metastore names (e.g., default.my_table). If your DataHub instance ingests tables via a Unity Catalog source, those tables are recorded with their three-part catalog names (e.g., my_catalog.default.my_table). The names never match, so the job's Lineage tab shows zero upstream and downstream tables, and the tables appear only under "Show Hidden Edges."

Use the following Spark properties to align the agent's emitted names with the Unity Catalog ingestion naming convention:

spark.datahub.metadata.dataset.hivePlatformAlias=databricks
spark.datahub.platform.databricks.platformInstance=<your-unity-catalog-name>
spark.datahub.flow_name=<descriptive-name-for-this-workflow>

• hivePlatformAlias=databricks — instructs the agent to emit table URNs under the databricks platform instead of hive, matching what the Unity Catalog ingestion source produces.
• platformInstance — set to the Unity Catalog name (the top-level catalog identifier) so the three-part name is reconstructed correctly.
• flow_name — optional but recommended; gives each workflow its own named entry in DataHub instead of defaulting to the cluster name and shell session identifier.
• Leave spark.datahub.legacyLineageCleanup.enabled unset when using these properties, to avoid overwriting lineage already recorded by the ingestion source.
• Note: Native catalog-name propagation support in the agent is under active development. These alias settings may need to be removed or adjusted once the upstream fix ships. Check release notes when upgrading.

Additionally, ensure the Unity Catalog used by the Databricks jobs is visible to the ingestion source workspace and that the ingestion service principal has been granted access to it. If the catalog is isolated to a different workspace, either bind it to the ingestion workspace or add a separate Unity Catalog ingestion source for that workspace.

Step 7 — Validate end-to-end

1. Restart the cluster and confirm no duplicate acryl JARs:

   %sh
   ls -la /databricks/jars | grep -iE 'acryl|scala-library'

2. Run a simple query to confirm the driver starts cleanly:

   SELECT 1

3. Run a CREATE TABLE AS SELECT that reads from and writes to known Unity Catalog tables:

   CREATE OR REPLACE TABLE <catalog>.<schema>.<target_table> AS
   SELECT * FROM <catalog>.<schema>.<source_table>

4. In DataHub, navigate to the data job entity and verify that upstream and downstream table links are present on the Lineage tab.

Additional Notes

• Supported runtimes: DBR 17.3 LTS (Spark 4.0, Scala 2.13, Java 17) is the highest Databricks Runtime currently validated with acryl-spark-lineage. DBR 19.x (Spark 4.2) is not yet supported due to an IncompatibleClassChangeError caused by a Spark 4.2 API change (CatalogManager changed from a class to an interface). Support for Spark 4.2 is under development.
• JAR naming: The Maven coordinate uses a colon separator (io.acryl:acryl-spark-lineage_2.13:1.7.0.14), but the filename on disk uses a hyphen before the version (acryl-spark-lineage_2.13-1.7.0.14.jar). Using the colon form as a filename in an init script results in a silent copy failure and a subsequent ClassNotFoundException.
• Deprecated DBFS paths: Databricks has end-of-lifed DBFS as a primary storage location for init script artifacts. Prefer Databricks Volumes or direct Maven downloads in init scripts over dbfs:/ paths.
• Old artifact