Lakehouse architecture · Part two of two
The lakehouse catalogs have absorbed the glossary, the classification, the lineage and the marketplace. So what is an enterprise governance catalog for now — and why does none of them speak the one protocol the market converged on? Collibra and IBM's Knowledge Catalog, held to the same test.
Part one of this pair, Six Things Called Catalog, One Credential Underneath, ended at a waist: table truth in the lakehouse has converged on the Iceberg REST catalog, and everything above it — governance, semantics, distribution — has not converged on anything at all.
This article is about the products that live in that unconverged band and have done for a decade. Collibra, Alation, Atlan, Informatica, Microsoft Purview, IBM's Knowledge Catalog. The category has a positioning problem that nobody in it enjoys discussing, and it comes down to one sentence.
An enterprise governance catalog is an L4/L5 product with no presence at L2 and no enforcement engine of its own. Everything else follows from those two facts, including the things it is genuinely good at.
Start with the mechanism, because the marketing word covers two different things.
Collibra can enforce policy on exactly four platforms, through two separate products, and in every case by pushing policy into the platform's native engine. Collibra Protect supports the following data sources: AWS Lake Formation, BigQuery, Databricks, Snowflake
1, and its own documentation describes the mechanism as Edge capabilities translating the representation to actions toward the data source provider using their technology.
On Databricks the output is a native object: Data access standards created in Protect result in column-based policies on Databricks.
2
The newer Collibra Data Access — which appears to be the productised form of the Raito acquisition, though no Collibra document says so13 — covers BigQuery, Databricks and Snowflake plus Entra ID and Okta, and unlike Protect it reads existing native grants as well as writing them.4 Collibra warns that the two must not overlap: To prevent policy drift and inconsistent security, do not manage the same data sources with Data Access and Protect at the same time.
Note the seam that leaves: Data Access does not cover Lake Formation, so a Lake Formation customer cannot move to the newer product.
Three documented limitations shape the operating model rather than the feature list, and they are the ones to raise in a design review:
Protect can't import or synchronize policies created directly in the data source. Synchronization is one-way: you must define your policies (standards and rules) in Collibra.3 Anything an engineer sets natively is invisible to it.
Although Protect removes any masking or row filtering, users can still access the data until they manually revoke the access in Databricks or Snowflake.3
Now the part that makes the rest worth reading.
IBM's watsonx.data intelligence — the product formerly and still widely called IBM Knowledge Catalog — occupies the same slot in the model. Held to the same criterion, its enforcement reach is narrower than Collibra's, and it is not close.
Take the two documented scopes and put them side by side, because that is the only comparison the evidence actually supports. Collibra states its own: Protect supports the following data sources: AWS Lake Formation, BigQuery, Databricks, Snowflake.
1 IBM states its own: You can define IKC governance policies for Presto (C++), and Presto (Java) engines.
15 Four platforms on one side; two engines inside one IBM product on the other. Both sentences are the vendor's own, unhedged, and that is the whole comparison — four platforms against two engines.
The connector asymmetry sharpens it. IBM's metadata harvest reaches the major non-IBM platforms — Snowflake, Databricks, BigQuery, Redshift, Teradata, SAP. But the connector list that carries governance into watsonx.data is eleven sources long and contains none of those four: IBM Knowledge Catalog - watsonx.data integration supports the following connectors for governance:
Hive, Iceberg, Hudi, Delta Lake, Oracle, PostgreSQL, MySQL, SQL Server, Db2, Teradata, MongoDB.15 Breadth of cataloguing is not breadth of control, and the gap between the two lists is where that distinction lives.
only one of the following policy engines— Apache Ranger or IBM Knowledge Catalog.16 Ranger's service types cover Presto and Spark, including external Spark via an extension. Knowledge Catalog is documented for the two Presto engines only. Choosing IBM's governance catalog over the open-source policy engine costs you Spark enforcement. That is a real trade and IBM does not, as far as I can find, publish it as a comparison table.
Two vendors, the same shape. The gap between the wide band and the narrow one is where "we govern your data estate" quietly becomes "we catalogue your data estate".
Two more findings that belong in the same paragraph as the Collibra limitations, for symmetry. IBM documents its own enforcement defects, and they are not cosmetic: Attempt to download protected data assets is allowed
, and Masked data might be profiled when the data source is IBM watsonx.data.
17 And IBM's own API documentation uses the verb the whole category should be using: the Knowledge Catalog APIs let you define data protection rules
18. Define. Not enforce.
Here is the structural point, and it applies to both products equally.
L2 converged on the Iceberg REST catalog. Collibra has no Iceberg REST catalog integration and no Apache Polaris integration. Across its current product documentation and both 2026 release-note sets, the only appearance of Iceberg is as a value of a Snowflake attribute — the table type shown as 'Dynamic Table,' 'Hybrid Table,' 'Iceberg Table,' or 'Base Table.'
5 Collibra can record that a Snowflake table happens to be Iceberg. It cannot talk to an Iceberg catalog.
Its integration surface is overwhelmingly JDBC metadata harvest6 — even AWS Lake Formation is reached through Amazon Athena's JDBC driver rather than a Lake Formation API7 — with outbound push on exactly two platforms: Unity Catalog, which supports both inbound and outbound (in preview) metadata flows
8 and is gated behind a support request, carrying tags only; and Google Knowledge Catalog, where outbound went GA.9
IBM is in the identical structural position, and sounds as though it is not. IBM markets deep Iceberg, Unity and Polaris adjacency — but that capability belongs to the watsonx.data lakehouse query engine, which is a different product from the governance catalog. The Metadata Service implements selected APIs from the Iceberg REST Catalog and Unity Catalog Open API spec
19, and the federation to Snowflake Open Catalog and Unity Catalog is engine-level, done by Presto and Spark for query purposes.20 The governance layer reaches that estate through a connection configured with a hostname, a port and an engine ID. It sees tables through a connector surface, exactly like Collibra does.
For anyone running on-premises, this is the sharpest gap in the category, and it cuts both ways.
Ranger is the enforcement engine of most Hadoop-descended and Trino-based estates, and of watsonx.data whenever Ranger rather than Knowledge Catalog is elected. There is no Collibra-built Apache Ranger integration. No Ranger page exists in Collibra's product documentation. The route that does exist is a third-party partner listing whose own terms state that Your Master Agreement with Collibra for the Collibra Service DOES NOT apply to your use of the Partner Offerings.
10 The same partner supplies the Trino and Starburst equivalent. Neither listing shows a version or a supported Collibra release.
There is a matching lineage asymmetry on both sides, and it is almost comic. Collibra ingests metadata from Trino, Presto and Starburst — all three are on its certified connector list12 — but none of the three appears among the seventeen SQL dialects it parses for technical lineage.11 And IBM's lineage configurations, which run to thirty-two sources including Informatica, SSIS, Talend, SAS and Db2 for z/OS, do not include Trino or Presto either21 — Presto being the engine IBM's own lakehouse runs on. That second finding is weaker evidence than the first: it comes from the documentation site's navigation, corroborated in two separate collections, because the authoritative table refused retrieval. Collibra's gap is quoted from a dated page; IBM's is inferred from an index. Two governance catalogs, neither of which parses lineage for the query engine at the centre of the architecture they are governing.
One counterweight worth recording, because it is the only real bridge in either direction: IBM's Common Policy Gateway is an extensibility point, and a release note names Collibra as a candidate engine — a downloadable plugin, enabling seamless integration with any policy engine (for example, IBM Knowledge Catalog, Apache Ranger, Collibra)
22. But the gateway's own topic page narrows the list to two, and the plugin is something the customer writes. It is a socket, not a shipped integration.
Everything above collapses into one operating principle, and it is the sentence I would put on the wall of any team running an enterprise catalog next to a lakehouse.
Read the four boxes again and notice what they have in common: none of them is a feature gap. They are properties of the arrangement. You would get all four from a perfectly implemented product on both ends, because the arrangement puts a translation step between the place where intent is expressed and the place where it binds.
Which is why the useful questions in a vendor evaluation are not about the catalog's feature matrix at all. They are: what is the compilation interval, is it one-way or reconciling, what does the target refuse to express, and what monitors the compiler.
Five years ago the division of labour was clean. Metastores were dumb; the enterprise catalog held the glossary, the classifications, the lineage, the quality scores and the stewardship workflow. That division no longer holds on any of the three platforms.
| Capability | Unity Catalog | Snowflake Horizon | IBM estate | Still differentiated above? |
|---|---|---|---|---|
| Business glossary | Pages, Beta — governed definitions of business concepts23 | Semantic Views and Horizon context | Knowledge Catalog glossary | Only across estates. Within one platform, no. |
| Classification | Native, agent-driven, GA24 | Automatically discover and classify columns25 | Native classes and profiling | No. |
| Lineage | Computed natively; Collibra reads it from system tables rather than parsing11 | Column-level, plus OpenLineage feeds25 | Dedicated lineage engine — but not for Presto21 | Only for the seams — dbt, BI tools, cross-platform hops. |
| Data quality | Expectations and monitors | Native | Native rules and scoring | Contested. |
| Marketplace | Databricks Marketplace | Internal Marketplace: Discover and share governed data products across teams without copying data.25 | Data Product Hub | Only across estates. |
| Enforcement | Native, in the query path | Native, in the query path | Ranger or Knowledge Catalog, in the query path | Never — the category delegates by construction. |
Collibra's own positioning concedes the shift, and the verb is the tell. Announcing its Databricks partner award in June 2026, the company wrote that Unity Catalog gives Databricks customers a powerful governance foundation for data and AI assets across the Lakehouse
and that Collibra complements that foundation, enriching Unity Catalog with the broader enterprise context layer.
14 Complements. Enriches. Not governs.
IBM does the same thing in its own documentation, and more revealingly, because it is technical prose rather than a press release. IBM describes Snowflake Open Catalog is a unified governance solution for Apache Iceberg tables in Snowflake
and Unity Catalog as a unified governance solution for data and AI assets in Databricks
20 — in the course of documenting how to federate to them. IBM is calling its competitors' catalogs governance solutions while positioning its own governance catalog above the same estate.
This is not a decline story. It is a change of job. But it is worth naming, because a great many architecture diagrams still show the enterprise catalog sitting on top as though it were the control plane, and it is not one.
Four things survive the encroachment, and they are the four to buy for:
discover data assets, explore lineage, search the Business Glossary27. Being the cross-platform context an agent queries is a defensible position, whatever one thinks of the category.
Collibra is not SaaS-only: Collibra Platform Self-Hosted is actively developed, and air-gapped installation is documented, with FIPS mode mandatory in that mode.28 But the self-hosted product is materially reduced, and the published Edge capability list for it covers ingestion, profiling, classification, sampling, synchronisation and the data-quality connector — with neither Protect nor Data Access on it.
I want to be careful. I could not find a page stating in words that policy enforcement is unavailable on self-hosted Collibra; the capability list omits it, which is strong but indirect. Treat "air-gapped Collibra cannot push policy" as a well-supported inference to verify with the vendor, not as a documented fact. If it holds, it decides any sovereign design: you get the authoring plane and you do not get the compilation step.
For data-residency reviews, one more: technical lineage on the SaaS path is processed by Collibra-operated regional cloud services — Collibra Data Lineage service instances are cloud-hosted endpoints that process and store technical lineage metadata.
29 Metadata leaves the network, not data. A self-hosted air-gapped lineage package exists, so this is a SaaS-path constraint rather than an absolute one.
Which brings the two articles together.
At the waist, the market has a contract. An engine written against the Iceberg REST specification reads tables owned by any of the three platforms, and that convergence happened in about two years. Above the waist there is nothing of the kind. There is no interchange standard for policy. OpenLineage covers lineage. The Open Data Contract Standard covers data products. Apache Ossie — the Open Semantic Interchange, renamed on entering the Apache Incubator in July 202630 — is attempting semantics. Nothing covers a masking rule.
That absence is the whole explanation for this article. It is why enforcement is a per-platform push-down rather than a protocol. It is why the compilation step is bespoke, unmonitored and invisible to both ends. It is why Collibra reaches four platforms and IBM reaches its own Presto engines, rather than both reaching everything that speaks a common contract. And it is why the category's honest self-description has shifted from governs to complements.
The Iceberg project has looked at putting fine-grained access control into the REST specification twice, and closed both attempts as not planned. The current proposal puts the enforcement obligation on the client — a reader must enforce projections on the columns it is actually reading
31 — and the reasoning on the project's own list is that proof is unaffordable, so trust will have to do. That is a coherent position. It also means the layer above the waist is not going to converge by protocol any time soon.
And here the two halves of this pair meet, which is the thing I did not see until both were written. Part one ended on a credential: to let an engine read the bytes, the catalog hands out a storage token scoped to a path prefix and carrying nobody's identity, at which point a column mask and a row filter become not merely unenforced but unevaluable. This article ends on an absence: above the waist there is no interchange contract for policy at all. They are the same fact seen from two ends. Below the waist, openness was bought by handing out a credential that fine-grained policy cannot survive. Above it, no standard emerged to carry that policy across perimeters — because the layer that would have had to transport it is precisely the layer the open protocol declined to model. The lakehouse got one contract, and it got it at the exact altitude where governance stops being expressible.
And whichever product you pick, the question that decides whether the design works is the same one, and neither vendor's feature matrix answers it: what compiles authored intent into enforced policy, how often, and how do you find out when it stops?
Every source below was retrieved on 31 August 2026. Press releases and product marketing pages are marked as such and cited only for positioning, never for feature status. Where IBM's published documentation site refused automated retrieval, IBM's own documentation source repository was used and the provenance is flagged; where a finding rests on a page's navigation rather than its body text, that is flagged too.
Protect supports the following data sources: AWS Lake Formation, BigQuery, Databricks, Snowflake.; and on mechanism, Edge capabilities
translating the representation to actions toward the data source provider using their technology.Dated 30 June 2026. productresources.collibra.com ↩
Data access standards created in Protect result in column-based policies on Databricks. Column-based policies are applied directly to the columns on Databricks.Dated 4 May 2026. productresources.collibra.com ↩
Protect can't import or synchronize policies created directly in the data source. Synchronization is one-way;
Although Protect removes any masking or row filtering, users can still access the data until they manually revoke the access in Databricks or Snowflake.; hourly synchronisation by default. Dated 26 June 2026. productresources.collibra.com ↩
These access controls are automatically enforced in your underlying data sources; supported sources
BigQuery, Databricks, Snowflakeplus Entra ID and Okta;
Both inbound and outbound; and
To prevent policy drift and inconsistent security, do not manage the same data sources with Data Access and Protect at the same time.Dated 11 August 2026. productresources.collibra.com ↩
'Dynamic Table,' 'Hybrid Table,' 'Iceberg Table,' or 'Base Table.'), Semantic Views ingestion reaching GA, and the Unity Catalog outbound gate:
This feature is in public preview and not enabled by default. Contact Collibra Support to activate it for your instance.2026.06 · 2026.05 ↩
Ingest data from AWS Lake Formation: Download the JDBC driver for Amazon Athena.productresources.collibra.com ↩
The integration supports both inbound and outbound (in preview) metadata flows to keep your catalog in sync with Databricks Unity Catalog.Dated 21 August 2026. productresources.collibra.com ↩
It supports both inbound and outbound metadata flows; and the scope limit,
The Knowledge Catalog integration ingests metadata only; it does not create technical lineage.Dated 12 August 2026. productresources.collibra.com ↩
Partners create, own and are responsible for their Partner Offerings… Your Master Agreement with Collibra for the Collibra Service DOES NOT apply to your use of the Partner Offerings.The same partner lists a Trino/Starburst equivalent. Neither shows a version or supported Collibra release. marketplace.collibra.com ↩
the lineage system tablesrather than parsed. Also records that
The CLI lineage harvester reached its end of life on July 31, 2026.Dated 29 July 2026. productresources.collibra.com ↩
Collibra complements that foundation, enriching Unity Catalog with the broader enterprise context layer.collibra.com — vendor press release, cited for positioning ↩
You can define IKC governance policies for Presto (C++), and Presto (Java) engines.and
IBM Knowledge Catalog - watsonx.data integration supports the following connectors for governance:followed by Hive, Iceberg, Hudi, Delta Lake, Oracle, PostgreSQL, MySQL, SQL Server, Db2, Teradata, MongoDB. IBM documentation source repository — ibm.com/docs refuses automated retrieval; repository HEAD dated 26 May 2026 ↩
You can only integrate with one of the following policy engines starting with watsonx.data version 2.1.; Ranger service types cover
tables used by Presto engine in watsonx.dataand, via
Hadoop SQL, the Spark engine. IBM documentation source repository ↩Attempt to download protected data assets is allowed;
Masked data might be profiled when the data source is IBM watsonx.data. The page opens
The following known issues and limitations apply to IBM Knowledge Catalog and to watsonx.data intelligence.ibm.com/docs — re-retrieved and both quotations re-confirmed verbatim on 31 August 2026 ↩
You can use the IBM Knowledge Catalog as a Service APIs to establish business vocabulary, import and enrich data assets, analyze data quality, define data protection rules, and more.cloud.ibm.com/apidocs ↩
Metadata Service implements selected APIs from the Iceberg REST Catalog and Unity Catalog Open API spec.IBM documentation source repository ↩
Snowflake Open Catalog is a unified governance solution for Apache Iceberg tables in Snowflake.and
Databricks Unity Catalog is a unified governance solution for data and AI assets in Databricks.— with federation performed by the Presto and Spark engines through the Iceberg REST Catalog API. Snowflake page · Databricks page — IBM documentation source repository ↩
A new lightweight CPG is now available as a downloadable plugin, enabling seamless integration with any policy engine (for example, IBM Knowledge Catalog, Apache Ranger, Collibra).The Common Policy Gateway topic page narrows the named list to two engines and the plugin is customer-written. IBM documentation source repository · gateway plugin page ↩
A Page is a governed, authoritative definition of a business concept— Beta. docs.databricks.com ↩
Databricks Data Classification uses an agent to automatically classify and tag tables in your catalog.docs.databricks.com ↩
Automatically discover and classify columns.;
Column-level lineage across Snowflake, external databases, BI tools, and OpenLineage feeds.;
Internal Marketplace: Discover and share governed data products across teams without copying data.docs.snowflake.com ↩
AI Command Center helps you to monitor and manage your AI landscape from a centralized dashboard.; it
replaces the fragmented product pages of AI Governance. Dated 3 August 2026. productresources.collibra.com ↩
The Collibra MCP server connects your AI assistant to governed metadata and business context in Collibra. Through it, an assistant can discover data assets, explore lineage, search the Business Glossary.Dated 26 June 2026; OAuth support GA per release notes 2026.06. productresources.collibra.com ↩
The Zarf package method is only supported if you have an Air-gapped Collibra Platform and environment.;
For Air-gapped environments, FIPS mode is mandatory. This page carries the Edge capability list on which neither Protect nor Data Access appears — the basis for the inference stated in the text. productresources.collibra.com · self-hosted overview ↩
Collibra Data Lineage service instances are cloud-hosted endpoints that process and store technical lineage metadata.Eleven regional endpoints. Dated 1 April 2026. productresources.collibra.com ↩
The project has been accepted into the Apache Incubator under a new name: Apache Ossie (Incubating)— the Open Semantic Interchange rename. snowflake.com ↩
A reader must enforce projections on the columns it is actually reading.Two earlier attempts, issues 10909 and 14187, were closed as not planned. patch-diff.githubusercontent.com ↩
Stated limits. Three claims rest on absence rather than statement and are marked as inferences in the text: that policy enforcement is unavailable on self-hosted Collibra; that neither Collibra nor IBM's governance catalog has an Iceberg REST or Apache Polaris integration; and that IBM does not push policy into non-IBM native engines. Each rests on a full read of the reachable product documentation, but absence cannot prove a page does not exist. Separately, a Snowflake–Collibra bi-directional integration was announced in June 2026 with no corresponding documentation or release-note entry, so it should be read as announced, not shipped. IBM's published documentation site refuses automated retrieval, so IBM findings come from its own documentation source repository, whose HEAD is dated 26 May 2026. Corrections are welcome and will be applied with the date attached.
David Leconte is a Customer Success Engineer in the Data & AI team at IBM France, covering Horizon Customers. He writes about lakehouse architecture, retrieval systems and the parts of data engineering that resist being tidied up. Corrections and correspondence: linkedin.com/in/davidleconte.
Part one: Six Things Called Catalog, One Credential Underneath.
Disclaimer. The postings on this site are my own and do not necessarily represent IBM’s positions, strategies or opinions. This article reflects the author’s own analysis and is not an IBM publication.
Trademarks. IBM, watsonx and watsonx.data are trademarks or registered trademarks of International Business Machines Corporation. Databricks and Unity Catalog are trademarks or registered trademarks of Databricks, Inc. Snowflake and Snowflake Horizon are trademarks or registered trademarks of Snowflake Inc. Collibra is a trademark or registered trademark of Collibra NV. Apache, Apache Iceberg, Apache Polaris, Apache Ranger and Apache Spark are trademarks or registered trademarks of the Apache Software Foundation. All other marks are the property of their respective owners. Their use here is nominative and descriptive; no affiliation, sponsorship or endorsement is implied.
© 2026 David Leconte. All rights reserved. Quotations from third-party documentation remain the property of their publishers and are reproduced, in short form and with attribution, for the purposes of analysis, criticism and commentary.