OpenMetadata
Overview
OpenMetadata is an open-source metadata platform that unifies data discovery, governance, observability, and lineage in a single catalog. In Ilum, OpenMetadata acts as the central place where every data asset produced by the platform — Hive Metastore tables, Delta Lake tables on S3/MinIO, Kyuubi-served queries, Superset dashboards, and Airflow DAG runs — is registered, profiled, classified, and connected by end-to-end OpenLineage edges.
Unlike Marquez (which Ilum still ships for raw OpenLineage event capture), OpenMetadata is a governance and discovery layer: catalogs, glossaries, domains, owners, tier classifications, PII tags, audit trails, column-level lineage, query usage, and role-based access — all driven from a single REST/GraphQL API with a fully featured UI.
OpenMetadata in the Ilum Stack
Ilum deploys OpenMetadata as a first-class sub-chart of the AIO release alongside its dependencies (OpenSearch for search, PostgreSQL for the catalog DB, and a Kubernetes-native ingestion pipeline runner). It is exposed inside the unified Ilum UI under the Modules → Data Governance section.
The module is reverse-proxied at /external/openmetadata/ by the same Nginx that fronts Superset, Airflow, MLflow, and the rest of the modules — same-origin embedding works out of the box (X-Frame-Options and CSP are pinned off in the chart so the iframe renders inside Ilum without console noise).
What OpenMetadata Brings to Ilum
- Unified catalog across engines — every Hive/Kyuubi/Delta table, every Superset chart and Airflow DAG appears in one searchable place.
- End-to-end lineage — Spark jobs running through Ilum Core emit OpenLineage events directly to OpenMetadata; HMS tables, Delta-on-S3 tables, Superset charts and Airflow tasks all become connected nodes on the same lineage graph.
- Governance primitives — domains, owners, tier, glossary terms, PII auto-classification, and SQL-based data quality tests.
- Profiler & sample data — column statistics and sampled rows for every Hive table (when the profiler is enabled).
- Usage analytics — most-queried tables, top users, cost hotspots — driven by Kyuubi/Hive query history (opt-in; disabled by default, see Troubleshooting).
Architecture
┌─────────────────────────────────────────┐
│ Ilum UI (Nginx) │
│ /external/openmetadata/ │
└───────────────────┬─────────────────────┘
│
┌─────────▼─────────┐
│ OpenMetadata │ ← OpenSearch (search index)
│ API + UI 8585 │ ← PostgreSQL (catalog state)
└──┬──────────┬──┬──┘
OpenLineage events│ REST│ │K8s Job runner
(HTTP transport) │ ingest │ │ (in-cluster pods)
│ │ │
┌────────┬─────────────┼──────────┼──┼──────────────┬──────────┐
│ │ │ │ │ │ │
Spark Jobs Airflow Hive Metastore Kyuubi Delta-on-S3 Superset
(ilum-core) (DAGs) (HMS Postgres) (Thrift) (MinIO logs) (charts)
Three complementary mechanisms feed metadata into OpenMetadata:
- Push (OpenLineage) — Spark jobs launched by
ilum-coreemit OpenLineage events through the composite transport to both Marquez and OpenMetadata's/api/v1/openlineage/lineageendpoint. OpenMetadata auto-creates the corresponding pipeline, dataset, and column-level lineage entities. - Pull (Ingestion pipelines) — A bootstrap Helm Job registers each connector and creates scheduled ingestion pipelines (metadata, profiler, auto-classification, dbt). Pipelines run as Kubernetes Jobs spawned by OpenMetadata's K8s pipeline client — no separate Airflow installation required.
- Enrich (Delta enricher CronJob) — A lightweight CronJob reads Delta Lake
_delta_log/directly from S3 and patches Delta-specific facets (protocol versions, file counts, partition and Z-Order columns) onto the correspondinghive.*table entities. It replaces the memory-hungry native DeltaLake connector — see Delta Lake metadata via the enricher below.
Ilum ships a small fork of the OpenMetadata server image (ilum/openmetadata:, pinned in helm_aio/values.yaml — see that file for the exact tag) built on top of the official 1.12.5 base. The fork recompiles two classes to fix OpenLineage auto-create reliability: OpenLineageEntityResolver sets updatedAt/updatedBy on auto-created entities (without it PostgreSQL silently rejects the INSERT and the lineage edge is dropped), and OpenLineageResource merges the existing edge's richer LineageDetails before re-inserting (so the periodic backfill never clobbers column-level lineage or SQL written by dbt ingestion). The patch tracks upstream issue open-metadata/OpenMetadata#27548 and is removed once the fix ships in an OpenMetadata release.
Enabling OpenMetadata
OpenMetadata is heavyweight (OpenMetadata server, OpenSearch, a dedicated database, a bootstrap Job, and the delta-enricher / openlineage-backfill CronJobs). The integration is controlled by the gates below. Enable it by setting them at install or upgrade time:
helm upgrade \
--set openmetadata.enabled=true \
--set openmetadata-dependencies.enabled=true \
--set openmetadataBootstrap.enabled=true \
--set ilum-core.job.openLineage.openmetadata.enabled=true \
--set global.lineage.enabled=true \
--reuse-values ilum ilum/ilum
The flags map to:
| Flag | What it does |
|---|---|
openmetadata.enabled | Deploys the OpenMetadata server (ilum-openmetadata, port 8585) |
openmetadata-dependencies.enabled | Deploys OpenSearch (the search backend) |
openmetadataBootstrap.enabled | Runs the post-install Job that registers services and pipelines |
ilum-core.job.openLineage.openmetadata.enabled | Wires the ilum-core OpenLineage composite transport to OpenMetadata |
global.lineage.enabled | Deploys Marquez, the OpenLineage backbone the integration relies on |
After the upgrade, OpenMetadata is reachable at:
- Inside Ilum UI: Modules → Open Metadata → Manage, or directly at
/external/openmetadata/ - Default credentials:
[email protected]/admin(single source of truth:openmetadataBootstrap.omUser/omPassword)
Access gate (SEC-01)
The /external/openmetadata/ route is deny-by-default. Nginx renders a gate that returns 403 unless the request carries an access_openmetadata=true cookie:
map $http_cookie $access_openmetadata {
default 0;
~*access_openmetadata=true 1;
}
# inside location /external/openmetadata/
if ($access_openmetadata = 0) { return 403; }
The Ilum UI sets that cookie automatically when a logged-in user with the OPENMETADATA_READ permission opens the module — the React app decodes the session JWT, maps each granted module permission to its cookie, and writes access_openmetadata=true; path=/; max-age=86400. The built-in ADMIN, USER, DATA_ENGINEER, and DATA_SCIENTIST roles all carry OPENMETADATA_READ, so the default users work with no extra configuration — open the module and the embedded UI loads.
The bootstrap credentials live in a Kubernetes Secret named ilum-openmetadata-admin (keys username, password), consumed by the bootstrap Job and the Airflow workers through valueFrom.secretKeyRef. The default password is admin; rotating it is described under Configuration Reference.
What the Bootstrap Job Registers
The post-install Job (bootstrap-openmetadata.sh) waits for OpenMetadata to be ready, authenticates as the admin user, and idempotently provisions five connectors. Re-running the upgrade re-applies the same configuration without duplicating entities — every call uses PUT (upsert) for services and a GET → POST pattern for pipelines.
| Service | Connector | What it provides |
|---|---|---|
Hive Metastore (hive) | Hive — HiveServer2 (Kyuubi :10009), Thrift only | Single canonical database service. The Hive connector owns table metadata, profiler, auto-classification, and dbt ingestion. All schema reads go through Kyuubi DESCRIBE — no direct HMS PostgreSQL connection (see the note below). |
Delta Lake on S3 (deltalake-s3) | DeltaLake (S3 storage service) | Registered as a catalog-tree placeholder and on-demand UI target. The native ingestion pipeline is disabled by default (nativeIngestionEnabled: false) because the connector embeds a JVM Spark runtime and OOM-kills at the 4Gi pod limit; Delta facets are delivered onto the hive.* tables by the Delta enricher CronJob instead. |
Superset (superset) | Superset dashboards | Charts, dashboards, and their underlying SQL — feeds the lineage graph downstream of tables. |
OpenLineage (openlineage) | Pipeline service | Receives /api/v1/openlineage/lineage events from Spark and auto-creates Pipeline entities. |
Iceberg on Nessie (iceberg-nessie) | Iceberg — REST catalog (Project Nessie /iceberg/) | Opt-in, off by default. Catalogs Apache Iceberg tables versioned in Project Nessie through Nessie's Iceberg-REST endpoint. Enabled only when both nessie.enabled and openmetadataBootstrap.services.iceberg.enabled are set — see Iceberg tables via Project Nessie below. Independent of the Hive service; both can run together. |
hive service instead of separate hive and kyuubi?Earlier Ilum releases registered Kyuubi as its own ilum-sql database service in addition to hive — both pointed at the same Kyuubi backend, which produced duplicate table FQNs in the catalog tree and split lineage edges across two services. Since 2026-04, the bootstrap consolidates everything onto a single hive service. One service, one canonical FQN, no duplicates.
Each enabled service has at least one ingestion pipeline scheduled by the in-cluster Kubernetes Job runner.
The Hive service is registered with the HiveServer2 Thrift endpoint only (ilum-sql-thrift-binary:10009, Kyuubi) and no metastoreConnection block. A direct HMS PostgreSQL connection points OpenMetadata at the COLUMNS_V2 table, which is empty for Spark-native Parquet tables (CREATE TABLE … USING parquet stores the schema in the file footer, not in HMS). The connector would then ingest a stub col ARRAY column and auto-classification would fail with UNRESOLVED_COLUMN. Without metastoreConnection, OpenMetadata falls back to its pyhive Thrift connector and resolves real columns through Kyuubi DESCRIBE. The services.hive.metastore.* keys in values.yaml are vestigial — they are no longer read into the service body.
Iceberg tables via Project Nessie
Ilum can catalog Apache Iceberg tables that are version-controlled in Project Nessie, so the same tables a Spark job branches and commits through Nessie become discoverable, classifiable, and lineage-connected assets in OpenMetadata. This is an opt-in capability that is off by default — a stock install registers the Hive service only and catalogs zero Iceberg tables.
OpenMetadata 1.12.5 has no native Nessie connector. Ilum instead registers an iceberg-nessie database service that uses OpenMetadata's Iceberg connector pointed at a REST catalog — Nessie's Iceberg-REST base URI (http://ilum-nessie:19120/iceberg). The REST catalog hands the ingestion worker only the pointer to each table's current metadata.json; the worker then reads the Iceberg metadata files themselves (metadata.json, manifest lists, manifests) directly from object storage to resolve schema, partition spec, and snapshots. This is why the worker is given the cluster's shared object-storage credentials — they are load-bearing for metadata ingestion, not just for reading row data. A reader identity that cannot read s3://ilum-data/nessie_catalog/ registers the service but catalogs zero tables.
The service body carries a non-secret token sentinel (nessie-rest-no-auth). OpenMetadata 1.12.5's Iceberg connector models the catalog connection as an ordered union that tries the Hive shape before the REST shape, and a bare {uri, fileSystem} body matches both — so the worker would otherwise build a Hive (Thrift) catalog against the REST URI and time out. A present token is accepted only by the REST shape, forcing the correct binding. Nessie runs with authType: NONE here, so the bearer header the client sends is ignored; replace the sentinel with a real token (via the OpenMetadata UI — operator overrides are preserved across bootstrap re-runs) only if Nessie authentication is ever enabled.
How it works
- Catalog source — Nessie's chart-managed
catalog:block exposes the Iceberg-REST API for thenessie_catalogwarehouse (s3://ilum-data/nessie_catalog/). The OpenMetadata service addresses it by the symbolic warehouse namenessie_catalog, not an S3 path; Nessie resolves the name to its storage location server-side. - Reference (branch) — the Nessie reference is selected by the warehouse, not by the REST URI path. The
restUriis the bare base (http://ilum-nessie:19120/iceberg, no ref); Nessie maps the warehouse name to a|prefix that the REST client uses for every call, and the default warehousenessie_catalogresolves to refmain. OpenMetadata catalogs that one reference; Nessie's branch/tag/commit timeline has no equivalent in OpenMetadata's data model, so only the tables visible on that ref are ingested, and merges/commits do not appear as lineage. To catalog a different ref, define a Nessie warehouse pinned to it and setopenmetadataBootstrap.services.iceberg.warehouse/catalogNameaccordingly — do not put the ref in the URI path (a ref in the path is doubled against the prefix and breaks namespace listing). - Deleted tables — the ingestion pipeline runs with
markDeletedTables: false, because Nessie's branch-scoped delete semantics do not map onto OpenMetadata's "table no longer present" check and would otherwise flap tables on every commit. - Coexistence — the Iceberg service is independent of the Hive service. Enabling Iceberg does not change Hive/Delta registration; both run side by side.
Enabling
This integration is off by default and there is no zero-cost background mode: turning it on starts the Nessie service (its own pod plus a Postgres database), so enable it only when Iceberg cataloging is actually wanted. Both flags below must be set — the effective gate ANDs them, so a stock install (Nessie off) registers Hive only and never starts a Nessie pod.
The fastest path is a single override at install or upgrade time:
helm upgrade ilum ./helm_aio \
--set nessie.enabled=true \
--set openmetadataBootstrap.services.iceberg.enabled=true
Or, equivalently, in a values overlay:
nessie:
enabled: true # starts Nessie + its Iceberg-REST catalog (adds a pod + DB)
openmetadataBootstrap:
services:
iceberg:
enabled: true # default false; the gate ANDs this with nessie.enabled
# schedule: "15 */2 * * *" # ingestion cadence (default); empty disables the pipeline
The S3-reader credentials, endpoint, region, and warehouse name default to the shared object-storage configuration and rarely need overriding. The full key list is under openmetadataBootstrap.services.iceberg.* in values.yaml.
The Nessie chart's Iceberg-REST catalog reads and writes table data with a static S3 credential reference pinned to the default MinIO Secret. On a RustFS (or other non-default) object-storage backend, override nessie.catalog.storage.s3.defaultOptions.accessKeySecret so Nessie and the OpenMetadata ingestion worker resolve the same backend identity — otherwise the worker authenticates as a different principal and reads zero rows. See UPGRADE_NOTES.md for the exact keys.
This section describes a newly added, opt-in integration. The configuration keys and registration flow are validated against the chart, but the end-to-end ingestion outcome (an Iceberg table written to Nessie appearing in the OpenMetadata catalog) is pending live verification on a running cluster. Treat it as a preview until that confirmation lands.
Domains and Hierarchy
The bootstrap Job creates a default domain (ilum) and, when defaultDomainStructure: hierarchical is set, a per-registered-service subdomain (ilum.hive, ilum.superset, ilum.deltalake-s3, and ilum.iceberg-nessie when Iceberg is enabled). This gives every freshly registered asset a sensible governance bucket without forcing data stewards to drag-and-drop assets into domains by hand.
To collapse everything into one domain, switch to flat:
openmetadataBootstrap:
defaultDomain: "ilum"
defaultDomainStructure: "flat" # or "hierarchical" (default)
The bootstrap is non-destructive: if a service already has an explicit (non-inherited) domain in OpenMetadata, the assignment is left alone — user choice always wins over the default.
Governance Seeding
The bootstrap Job seeds a neutral governance skeleton so a fresh install does not present an empty Domains page: it PUT-upserts a default domain (ilum) and, optionally, an empty Data Product. Column-level classification (PII tags) and attaching a concrete table set to a Data Product are not done by the bootstrap — they are owned by the pipeline that produces the data (for example the banking demo DAG), which knows the real schema and can tag columns[].tags with an RFC-6902 JSON-Patch and call PUT /v1/dataProducts/{id}/assets/add against the tables it just created. Keeping PII seeding with the producing pipeline avoids the bootstrap guessing column names against tables that may not exist yet on a fresh install.
The governance skeleton is configured under openmetadataBootstrap.governance in values.yaml:
openmetadataBootstrap:
governance:
enabled: true
domain: "ilum"
dataProduct: "" # empty = no Data Product created
dataProductDescription: ""
Set governance.enabled: false to start OpenMetadata with no pre-seeded domain or Data Product.
AutoClassification runs with storeSampleData set to false. OpenMetadata classifies columns without materializing sample rows of (potentially sensitive) data into its own store. Re-enable sampling with HIVE_STORE_SAMPLE_DATA=true and cap the row count with HIVE_SAMPLE_DATA_COUNT.
The Job exits non-zero on a critical error (a never-ready server, or a 401/403 credential rejection), so Kubernetes reports it Failed and operators get a clear signal — it no longer reports green while leaving the catalog un-seeded. A run summary is written to stderr prefixed BOOTSTRAP_SUMMARY: (and to the ilum-om-bootstrap-status ConfigMap when kubectl is available in the image). To restore the legacy best-effort behaviour where every failure degrades to exit 0, set BOOTSTRAP_SOFT_FAIL=true.
Delta Lake metadata via the enricher
OpenMetadata 1.12.5's native DeltaLake-S3 connector embeds a JVM Spark runtime and OOM-kills at the 4Gi ingestion-pod limit on large tables. Rather than raise that limit, Ilum ships a small Delta enricher CronJob that reads each table's _delta_log/ directly from S3 via the Rust deltalake library and PATCHes the Delta-specific facets — protocol versions, file count, last commit, partition columns, and Z-Order columns — onto the matching hive.* table customProperties in OpenMetadata.
openmetadataBootstrap:
deltaEnricher:
enabled: true
schedule: "0 */6 * * *" # Delta facets change slowly; every 6h is ample
image: "ilum/delta-enricher:1.1" # pre-baked deltalake+psycopg2+boto3; tag pinned in helm_aio/values.yaml
omApiEndpoint: "http://ilum-openmetadata:8585/api"
The deltalake-s3 database service is still registered (it gives Delta tables a catalog-tree row and an on-demand UI target), but its scheduled ingestion pipeline is not created while services.deltalake.nativeIngestionEnabled is false. Set that flag to true (and raise the pipeline memory) only if the native S3 connector is specifically required.
Lineage from Spark Jobs
Spark jobs launched by ilum-core are configured by the chart to emit OpenLineage events through the composite HTTP transport. The relevant helm_aio/values.yaml block:
ilum-core:
job:
openLineage:
transport:
type: http
serverUrl: http://ilum-marquez:9555
endpoint: /api/v1/lineage
openmetadata:
enabled: true
url: http://ilum-openmetadata:8585
endpoint: /api/v1/openlineage/lineage
auth:
email: [email protected]
password: admin
facetsDisabled: "[spark_unknown;spark.logicalPlan]"
At runtime ilum-core does not use the literal password shown above for long-lived engines. By default it waits for a short-lived bot token (OPENMETADATA_TOKEN_LINEAGE, read from the ilum-om-bot-tokens Secret minted by the bootstrap Job) and authenticates the OpenLineage transport with that — auth.waitForBotToken: true and auth.waitForBotTokenFailClosed: true (both default) make core refuse to start rather than fall back to the 1-hour admin login that would expire on long-running engines. The email/password literal is only the best-effort fallback when the bot-token path is disabled. To point core at an externally-seeded token Secret, set ilum-core.job.openLineage.openmetadata.auth.botTokenExistingSecret / botTokenExistingSecretKey.
facetsDisabled pins the OpenLineage agent's facet filter to a stable allowlist; the default has drifted across releases and silently blocks lineage for PySpark Delta path-based writes (ContextFactory: Query execution is null). Setting the value explicitly here keeps lineage emission deterministic across OpenLineage jar bumps.
The composite transport sends each event to both Marquez (raw OpenLineage capture) and OpenMetadata (governance + cross-tool stitching). When OpenMetadata receives an event, it auto-creates the pipeline entity, the input/output dataset entities, and column-level edges between them.
Combined with the Hive, Delta, and Superset ingestion pipelines, this produces a single connected graph from raw S3 objects → Delta tables → Spark transformations → Hive/Kyuubi tables → Superset charts → Airflow DAG runs.
A Spark job that writes a Delta table only through HMS (without using S3 paths in the plan) won't produce an OpenLineage edge — that's an OpenLineage limitation, not Ilum's. Use the Delta Lake on S3 connector to surface those tables on the catalog side; for true lineage edges, prefer Spark plans that read/write S3 paths directly.
Service Detail: What You Get Per Asset
Click any registered service to see ingested entities, agent status, and platform-level metrics:
For each table, OpenMetadata exposes:
- Schema — columns, types, partition keys, and (for Delta-on-S3) protocol versions
- Sample data — a configurable number of rows surfaced by the profiler
- Profiler metrics — null counts, distinct counts, min/max/mean/stddev per column
- Auto-classification — PII, sensitive, and tier tags assigned by the classification pipeline
- Lineage — both upstream and downstream, table-level and column-level
- Queries — top SELECTs hitting the table, sourced from Kyuubi/Hive usage ingestion (opt-in; usage ingestion is disabled by default)
- Owners, domain, glossary terms, custom properties
Configuring a dbt Project for OpenMetadata
OpenMetadata understands dbt projects natively. Once configured, every dbt model becomes a first-class catalog entity with: model SQL (raw + compiled), ref()/source() lineage, dbt tests as Test Cases, tags from meta blocks, owners from meta.owner, and descriptions from persist_docs. Ilum ingests dbt metadata inline from the Airflow DAG, so the catalog is refreshed in the same run that produced the data.
Inline ingestion from the Airflow DAG (immediate, gating)
A dbt DAG runs metadata ingest-dbt directly after dbt run finishes. The artifacts (manifest.json, catalog.json, run_results.json) are passed straight to the OpenMetadata CLI in the same Airflow worker, so the catalog is updated before any downstream governance task executes.
To wire a dbt DAG into this path, follow the pattern below:
# 1. dbt run (e.g. via Cosmos)
dbt_run = DbtTaskGroup(...)
# 2. publish artifacts to S3 (per-run + latest/)
publish_artifacts = PythonOperator(
task_id="publish_artifacts",
python_callable=publish_dbt_artifacts,
op_kwargs={"prefix": "my_project_artifacts"},
)
# 3. native OM ingest from local manifest
om_dbt_ingest = PythonOperator(
task_id="openmetadata_dbt_ingest",
python_callable=ingest_dbt_artifacts_to_openmetadata,
op_kwargs={
"dbt_project_dir": "/opt/airflow/dags/dbt_my_project",
"service_name": "hive", # MUST be "hive" — single canonical FQN
},
)
dbt_run >> publish_artifacts >> om_dbt_ingest >> downstream_tasks
Key requirements:
| What | Where | Value |
|---|---|---|
| dbt service in OM | dbt project's dbt_project.yml vars.openmetadata_service_name | hive |
| Env override (optional) | DAG env | OPENMETADATA_DBT_DATABASE_SERVICE_NAME=hive |
| Artifact S3 layout | publish_dbt_artifacts helper | s3://ilum-data/ |
| OM CLI version | airflow worker image | matches OM server (metadata package pinned to 1.12.x) |
Adding a New dbt Project — Checklist
- Set the service name to
hivein your dbt project'sdbt_project.yml:vars:
openmetadata_service_name: hive
openmetadata_dbt_classification_name: dbt_tags
openmetadata_dbt_update_descriptions: true
openmetadata_dbt_update_owners: true - Publish artifacts to S3 at
s3://ilum-data/using a Python helper invoked from the DAG._artifacts/{ ,latest}/manifest.json - Inline ingestion — add
openmetadata_dbt_ingesttask to the DAG (gating; ensures downstream tasks see fresh OM state). - Verify — open the table entities in the OpenMetadata UI and confirm the dbt tab is populated (model SQL,
ref()/source()lineage, tests) on each model produced by the run.
Why inline?
Running the ingest in the same DAG, gated after dbt run, makes downstream DAG tasks deterministic — a governance or tagging task will not race ahead of the dbt metadata, because the catalog is already refreshed before those tasks execute.
Configuration Reference
All bootstrap parameters live under openmetadataBootstrap in helm/helm_aio/values.yaml. The values of omUser and omPassword are the canonical source for the ilum-openmetadata-admin Secret consumed by the bootstrap Job and Airflow workers — that is, for the client credentials only. They are not the source of truth for the password stored inside OpenMetadata, which is seeded at first boot and rotated separately (see the note below). The most useful knobs:
openmetadataBootstrap:
enabled: true
omServerUrl: "http://ilum-openmetadata:8585"
omUser: "[email protected]"
omPassword: "admin"
omMaxWait: "300" # seconds to wait for OM readiness
triggerPipelines: true # run ingestions immediately after deploy
defaultDomain: "ilum"
defaultDomainStructure: "hierarchical" # or "flat"
# Schedules are deliberately staggered (different minute offsets) to avoid
# several ingestion pipelines hitting the Kyuubi engine at the same instant.
services:
superset:
enabled: true
host: "http://ilum-superset:8088/external/superset"
username: "admin"
password: "admin"
schedule: "45 */2 * * *"
hive:
enabled: true
hostPort: "ilum-sql-thrift-binary:10009" # Kyuubi (HiveServer2) — Thrift only, no metastoreConnection
schedule: "5 */2 * * *"
# Profiler + Sample Data tab + Auto-Classification — three features in one flag.
# Default schedule is once per night to avoid Kyuubi engine overload.
profilerEnabled: true
profilerSchedule: "30 2 * * *"
autoClassificationSchedule: "35 */6 * * *"
# Usage ingestion. Disabled by default: OM 1.12.5 ships no Hive usageSourceClass,
# so the pipeline crashes deterministically (Kyuubi also does not fully expose
# Hive `query_log`). Re-enable only once the Kyuubi audit-log shipper lands.
usageEnabled: false
usageSchedule: "10 */4 * * *"
# Native dbt ingestion — one pipeline per dbt project, reads manifest.json from S3.
# Per-project knobs follow the pattern dbt{Enabled,Prefix,Schedule}.
deltalake:
enabled: true # registers the databaseService (catalog-tree row)
bucketName: "ilum-data"
prefix: "" # narrow to a sub-tree if needed
schedule: "" # empty = no scheduled ingestion pipeline
nativeIngestionEnabled: false # native S3 connector OOM-kills; enricher feeds hive.* instead
# Delta enricher CronJob — the canonical path for Delta facets (see section above).
deltaEnricher:
enabled: true
schedule: "0 */6 * * *"
image: "ilum/delta-enricher:1.1" # pre-baked deltalake+psycopg2+boto3; tag pinned in helm_aio/values.yaml
omApiEndpoint: "http://ilum-openmetadata:8585/api"
# Governance skeleton (default domain + optional empty Data Product). See "Governance Seeding".
# PII tags + asset-attach are seeded by the producing pipeline (e.g. the banking DAG), not here.
governance:
enabled: true
domain: "ilum"
dataProduct: ""
dataProductDescription: ""
To disable a connector, set
omPassword only sets the client credentials in the ilum-openmetadata-admin Secret. OpenMetadata 1.12.5 seeds the admin account ([email protected] / admin) into its own store on first boot, and the chart has no knob that rewrites the server-side password — so editing omPassword alone desynchronizes the clients from the server. To rotate: (1) change the password inside OpenMetadata first (UI → user profile → Change Password), then (2) set openmetadataBootstrap.omPassword to the new value and re-run helm upgrade, then (3) roll the bootstrap and Airflow pods. The ilum-core OpenLineage transport authenticates with a short-lived bot token (see Lineage from Spark Jobs), so it is unaffected by the admin-password rotation.
Troubleshooting
| Symptom | Likely cause | What to check | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
The embedded OpenMetadata UI keeps reloading / shows 403 | The access_openmetadata gate cookie is missing on the current origin | Reach OpenMetadata on the same host and port used to log in to Ilum (the cookie is per-origin). Confirm the logged-in user has the OPENMETADATA_READ permission. See Access gate (SEC-01). | ||||||||||||||||||||||||
| Bootstrap Job reports Failed | A critical error (server never ready, or 401/403 credential rejection) | Read the BOOTSTRAP_SUMMARY: line in the Job logs (or the ilum-om-bootstrap-status ConfigMap) for the error list. Fix the cause, or set BOOTSTRAP_SOFT_FAIL=true to fall back to best-effort. The password is base64-encoded by the Job before login — set the plain password in values. | ||||||||||||||||||||||||
All OM clients return 401 after a helm upgrade that changed omPassword | omPassword was rotated in values without first running changePassword inside OpenMetadata | omPassword sets only the client Secret, not the server-side password. Follow the rotation procedure (change in OpenMetadata UI first, then values). With the default BOOTSTRAP_SOFT_FAIL=false, the bootstrap Job exits non-zero on this 401, so the Helm hook reports the failure rather than hiding it. | ||||||||||||||||||||||||
Hive table shows a single col array column | A metastoreConnection was added back, so OM read the empty COLUMNS_V2 instead of DESCRIBE | Remove any metastoreConnection from the Hive service — OM must resolve columns through Kyuubi DESCRIBE (Spark-native Parquet schemas are not in HMS). The legacy fix-hms-delta-columns Helm Job (fixHmsDeltaColumns.enabled: false) is no longer required. Re-run the Hive metadata ingestion pipeline. | ||||||||||||||||||||||||
| Spark jobs run, no lineage in OM | OpenLineage transport not reaching /api/v1/openlineage/lineage | Confirm ilum-core.job.openLineage.openmetadata.enabled: true and that the auth credentials match the bootstrap user. Confirm the OM image is the patched fork pinned in helm_aio/values.yaml, not the stock openmetadata/server image — the stock image silently drops auto-created lineage entities. | ||||||||||||||||||||||||
deltalake-s3 tables have no Delta facets (protocol, file count, partitions) | The native ingestion pipeline is off and the enricher has not run yet | Delta facets are delivered by the deltaEnricher CronJob onto the hive.* tables, every 6h by default. Trigger the CronJob manually, or set services.deltalake.nativeIngestionEnabled: true (and raise pipeline memory) for the native connector. | ||||||||||||||||||||||||
Lineage edges missing on a deltalake-s3.* table | All OpenLineage events route to the canonical hive service | This is by design — deltalake-s3 is a metadata-only catalog view; lineage lives on the hive. |