Area: API Issues
Sub-Area: Schema Field URN Construction / OpenAPI Integration
Issue
When datasets are ingested from AWS Glue or Hive via a DataHub ingestion recipe, schema field paths are stored in a v2 format that includes version and type annotations as a prefix — for example, [version=2.0].[type=int].eff_dt — rather than the plain column name. If you create datasets via the OpenAPI using only the bare field name as the fieldPath, a subsequent recipe-based ingestion will overwrite those paths with the v2 format, breaking any schema-field URNs you have already constructed (for example, for attaching structured properties). This behavior differs across platforms: Glue and Hive produce v2-format paths, while Snowflake and Teradata store the plain column name with no prefix. Understanding why this difference exists, and how to construct v2-format paths correctly in both Python and Java, is necessary to ensure that OpenAPI-created datasets remain compatible with recipe-generated metadata.
You Might Be Asking
- Why does DataHub use the full
fieldPathinstead of just the field name in schema-field URNs? - Why do Glue and Hive fields have
[version=2.0].[type=...]prefixes but Snowflake and Teradata fields do not? - How do I construct a v2
fieldPathin Python or Java so it matches what the ingestion connector produces? - How do I attach structured properties to a Glue schema field using the Java SDK?
- What is the correct schema-field URN format for a Glue/Hive dataset column?
Solution
1. Why DataHub Uses fieldPath Instead of Field Name
Field names alone are not guaranteed to be unique within a schema. For nested types such as structs, arrays, and maps, the same name can appear at multiple levels of the type hierarchy. The fieldPath encodes both the field name and its structural position, making it unambiguous regardless of nesting depth. This is required for correct fine-grained lineage and structured property attachment.
2. Why Glue/Hive and Snowflake/Teradata Produce Different Formats
The difference comes from how each connector parses schemas internally:
- Glue and Hive: These connectors parse Hive-style DDL type strings (e.g.,
int,struct<a:string,b:int>) and convert them to an Avro schema internally. This conversion pipeline unconditionally emits v2-format fieldPaths for every field, including plain primitives. A simple integer column namedeff_dtbecomes[version=2.0].[type=int].eff_dt. - Snowflake and Teradata: These are SQL-based connectors that reflect schema directly from the database via SQLAlchemy. They write the
fieldPathas the plain column name with no Avro conversion step and no type prefix — for any column type, including complex types like VARIANT or ARRAY.
3. Hive/Glue Type-to-Avro Mapping Reference
The following mappings apply when constructing v2 fieldPaths for Glue/Hive columns. Note the two non-obvious mappings: bigint maps to long, and binary maps to string.
int,integer,tinyint,smallint→[type=int]bigint,long→[type=long]float→[type=float]double→[type=double]string,varchar,char→[type=string]boolean,bool→[type=boolean]binary→[type=string](binary maps to string, not bytes)struct<...>→[type=struct].[type=struct]at top level, with subfields chained belowarray<T>→[type=struct].[type=array].[type=T]at top levelmap<K,V>→[type=struct].[type=map].[type=V]at top level (key type is always string in Avro)
4. Python: Using the Built-in Utility
Use the get_schema_fields_for_hive_column utility from the DataHub Python SDK. This is the same function used internally by the Glue connector, so the fieldPaths it produces are guaranteed to match what ingestion stores.
from datahub.utilities.hive_schema_to_avro import get_schema_fields_for_hive_column
from datahub.emitter.mce_builder import make_dataset_urn, make_schema_field_urn
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.emitter.mcp_builder import MetadataChangeProposalWrapper
from datahub.metadata.schema_classes import (
SchemaMetadataClass,
OtherSchemaClass,
StructuredPropertiesClass,
StructuredPropertyValueAssignmentClass,
)
emitter = DatahubRestEmitter("https://<your-datahub-host>", token="<your-token>")
dataset_urn = make_dataset_urn("glue", "mydb.mytable", "PROD")
# Define columns as (name, hive_type) pairs
columns = [
("eff_dt", "int"),
("account_id", "string"),
("amount", "bigint"),
]
# Build SchemaField objects using the same utility as the Glue connector.
# This guarantees fieldPaths match what the recipe produces.
all_fields = []
for col_name, col_type in columns:
all_fields.extend(get_schema_fields_for_hive_column(col_name, col_type))
# Emit the dataset with schema
emitter.emit_mcp(MetadataChangeProposalWrapper(
entityUrn=dataset_urn,
aspect=SchemaMetadataClass(
schemaName="mytable",
platform="urn:li:dataPlatform:glue",
version=0,
hash="",
platformSchema=OtherSchemaClass(rawSchema=""),
fields=all_fields,
),
))
# Construct the field URN for a specific column.
# Call the utility directly rather than indexing into all_fields,
# because complex columns expand into multiple entries.
eff_dt_field = get_schema_fields_for_hive_column("eff_dt", "int")[0]
# eff_dt_field.fieldPath == "[version=2.0].[type=int].eff_dt"
field_urn = make_schema_field_urn(dataset_urn, eff_dt_field.fieldPath)
# field_urn == "urn:li:schemaField:(urn:li:dataset:(...),
# [version=2.0].[type=int].eff_dt)"
# Attach a structured property to the field
emitter.emit_mcp(MetadataChangeProposalWrapper(
entityUrn=field_urn,
aspect=StructuredPropertiesClass(
properties=[StructuredPropertyValueAssignmentClass(
propertyUrn="urn:li:structuredProperty:<your-property-qualified-name>",
values=[365.0], # float for number properties, str for string properties
)]
),
))
For Snowflake or Teradata, the fieldPath is always the plain column name — no utility or prefix needed:
dataset_urn = make_dataset_urn("snowflake", "mydb.schema.mytable", "PROD")
field_urn = make_schema_field_urn(dataset_urn, "eff_dt")
# field_urn == "urn:li:schemaField:(urn:li:dataset:(...),eff_dt)"
5. Java: Constructing v2 Field Paths
The Java SDK does not include an equivalent to Python's get_schema_fields_for_hive_column. Use one of the following two approaches.
Option A (simplest): Fetch stored fieldPaths from the API after ingestion
If the dataset has been ingested at least once, read the exact fieldPaths that DataHub has stored:
GET /aspects/<url-encoded-dataset-urn>?aspect=schemaMetadata&version=0
Parse response.fields[i].fieldPath. No construction is needed and there is no risk of mismatch.
Option B: Construct fieldPaths in Java before first ingestion
Use the following self-contained helper class, which implements the same rules as the DataHub Python source:
import java.util.*;
public class GlueFieldPathBuilder {
private static final Map<String, String> PRIMITIVE_MAP;
static {
Map<String, String> m = new HashMap<>();
m.put("int", "int");
m.put("integer", "int");
m.put("tinyint", "int");
m.put("smallint", "int");
m.put("bigint", "long"); // bigint → long, not bigint
m.put("long", "long");
m.put("float", "float");
m.put("double", "double");
m.put("string", "string");
m.put("varchar", "string");
m.put("char", "string");
m.put("boolean", "boolean");
m.put("bool", "boolean");
m.put("binary", "string"); // binary → string, not bytes
PRIMITIVE_MAP = Collections.unmodifiableMap(m);
}
/**
* Returns all DataHub fieldPaths for a Glue/Hive column.
* The first entry is always the top-level column path.
* For struct columns, additional entries follow for each nested subfield.
*/
public static List<String> getFieldPaths(String colName, String hiveType) {
String normalized = hiveType.toLowerCase().trim();
List<String> paths = new ArrayList<>();
if (isPrimitive(normalized)) {
paths.add("[version=2.0].[type=" + avroType(normalized) + "]." + colName);
} else {
collectPaths("[version=2.0].[type=struct]", colName, normalized, paths);
}
return paths;
}
private static void collectPaths(String prefix, String name,
String hiveType, List<String> out) {
if (hiveType.startsWith("struct<")) {
out.add(prefix + ".[type=struct]." + name);
String inner = hiveType.substring(7, hiveType.length() - 1);
for (String[] field : parseStructFields(inner)) {
collectPaths(
prefix + ".[type=struct]." + name,
field[0],
field[1].toLowerCase().trim(),
out
);
}
} else if (hiveType.startsWith("array<")) {
String elemType = hiveType.substring(6, hiveType.length() - 1).trim();
collectPaths(prefix + ".[type=array]", name, elemType, out);
} else if (hiveType.startsWith("map<")) {
// Avro maps always have string keys; only the value type appears in the path.
String[] kv = splitMapKV(hiveType.substring(4, hiveType.length() - 1));
collectPaths(prefix + ".[type=map]", name, kv[1].trim(), out);
} else {
out.add(prefix + ".[type=" + avroType(hiveType) + "]." + name);
}
}
private static String avroType(String hiveType) {
return PRIMITIVE_MAP.getOrDefault(hiveType, hiveType);
}
private static boolean isPrimitive(String hiveType) {
return !hiveType.startsWith("struct<")
&& !hiveType.startsWith("array<")
&& !hiveType.startsWith("map<");
}
// Parses "f1:T1,f2:T2" respecting nested angle brackets
private static List<String[]> parseStructFields(String inner) {
List<String[]> fields = new ArrayList<>();
int depth = 0, start = 0;
for (int i = 0; i < inner.length(); i++) {
char c = inner.charAt(i);
if (c == '<') depth++;
else if (c == '>') depth--;
else if (c == ',' && depth == 0) {
fields.add(splitFieldDef(inner.substring(start, i).trim()));
start = i + 1;
}
}
fields.add(splitFieldDef(inner.substring(start).trim()));
return fields;
}
private static String[] splitFieldDef(String fieldDef) {
int colon = fieldDef.indexOf(':');
return new String[]{
fieldDef.substring(0, colon).trim(),
fieldDef.substring(colon + 1).trim()
};
}
private static String[] splitMapKV(String kv) {
int depth = 0;
for (int i = 0; i < kv.length(); i++) {
char c = kv.charAt(i);
if (c == '<') depth++;
else if (c == '>') depth--;
else if (c == ',' && depth == 0) {
return new String[]{ kv.substring(0, i).trim(), kv.substring(i + 1).trim() };
}
}
return new String[]{ kv.trim(), "" };
}
}
Sample output from the helper:
GlueFieldPathBuilder.getFieldPaths("eff_dt", "int")
// → ["[version=