Data Governance with Unity Catalog in Databricks

Unity Catalog gives Databricks teams one governance layer for controlling access, discovering assets, tracking lineage, classifying sensitive data and monitoring activity across connected workspaces. However, enabling Unity Catalog does not automatically create a complete governance operating model.

Teams must still decide how catalogs should be organized, who owns each data domain, how identities enter the platform, which policies should apply and how those controls will be tested. Poor decisions at this stage can reproduce the same permission sprawl and unclear ownership that Unity Catalog is intended to resolve.

This guide explains how Unity Catalog works, where RBAC, ABAC and table-level policies fit, and what teams should validate before moving governed workloads into production.

What Unity Catalog Governs in Databricks

Unity Catalog is the unified governance layer for data and AI in Databricks. It provides a central place to manage access, discovery, lineage, sensitive-data classification, quality monitoring and auditing across governed assets.

Those assets can include:

• Tables, views and materialized views
• Volumes containing non-tabular files
• Functions and registered models
• Feature tables and other AI-related assets
• External data accessed through governed connections
• Data shared with other teams or organizations

Unity Catalog also creates a consistent governance model across workspaces connected to the same metastore. Policies do not need to be recreated independently in every workspace, provided the identity, workspace and catalog architecture has been designed correctly.

Unity Catalog should not be treated as the only security layer. Authentication, network controls, encryption, cloud permissions and privacy configuration remain separate responsibilities.

Databricks also distinguishes the governance of data and AI assets in Unity Catalog from runtime controls for AI endpoints, agents and AI traffic provided through Unity Gateway.

How the Unity Catalog Hierarchy Works

Most governed assets use a three-level namespace:

catalog.schema.object

A transaction table might therefore be referenced as:

prod.finance.transactions

The hierarchy contains four important levels:

• Metastore: The top-level container for Unity Catalog metadata and governance objects.
• Catalog: A major governance boundary, commonly representing an environment, business domain or combination of both.
• Schema: A logical grouping of tables, views, volumes, functions and models inside a catalog.
• Object: The table, view, volume, function, model or other governed asset.

Some objects sit outside this three-level namespace. Storage credentials, external locations, connections, recipients and shares are governed through Unity Catalog but are not addressed as catalog.schema.object.

Understanding this distinction matters because catalog and schema design affects ownership, privilege inheritance, workspace access and how easily teams can separate development from production.

Design Catalogs Around Governance Boundaries

There is no single catalog structure that fits every organization. A small data team may use catalogs to separate development and production, while a larger enterprise may need distinct catalogs for finance, marketing, operations and customer data.

Catalog model	When it fits	Main concern
By environment	Strong separation between development, testing and production	Business-domain ownership may be less visible
By business domain	Teams own and govern their respective data products	Environment separation requires additional controls
Hybrid model	Large organizations with several domains and environments	More naming, automation and administration complexity

A catalog should represent a meaningful governance boundary, not simply mirror every technical layer in the data platform. Creating separate catalogs for Bronze, Silver and Gold can become difficult when the same team needs to manage a complete domain across all three layers.

For teams using a medallion architecture in databricks, schemas can often organize processing layers while catalogs enforce broader domain or environment boundaries.

Workspace bindings provide another control. They can restrict a catalog, storage credential or external location to specified workspaces. This helps prevent a development workspace from accessing production assets even when a user belongs to a group with broader privileges.

Manage Baseline Access with Groups and RBAC

Unity Catalog privileges should normally be assigned to account-level groups rather than individual users. Groups can be managed in an identity provider and provisioned into the Databricks account, making onboarding and offboarding easier to control.

Service principals should be used for jobs, pipelines and automated applications. They should receive only the privileges required by their workloads instead of inheriting a broad administrator role.

A basic read grant can look like this:

GRANT USE CATALOG ON CATALOG prod TO `finance-analysts`;
 
GRANT USE SCHEMA
ON SCHEMA prod.finance
TO `finance-analysts`;
 
GRANT SELECT
ON TABLE prod.finance.transactions
TO `finance-analysts`;

These privileges perform different jobs:

• USE CATALOG allows the group to interact with objects inside the catalog.
• USE SCHEMA allows it to interact with objects inside the schema.
• SELECT provides read access to the specified table.

USE CATALOG and USE SCHEMA are traversal privileges. They do not independently provide access to every object below them. Reading a table normally requires both traversal privileges and SELECT on the table or an inherited SELECT grant.

Broader privileges can be granted at a catalog or schema level and inherited by current and future objects. That can simplify management, but it also increases the effect of an incorrect grant. Apply broad inheritance only where the intended access boundary is clear.

Choose Between RBAC, ABAC and Table-Level Controls

RBAC, ABAC, row filters, column masks and dynamic views solve related but different problems.

Requirement	Suitable mechanism
Give a department access to a catalog or schema	Group-based RBAC
Apply one policy across many classified tables	ABAC
Mask one sensitive column on a specific table	Table-level column mask
Restrict a table by region, tenant or department	Table-level row filter
Expose joined or transformed protected data	Dynamic view
Restrict production assets to production workspaces	Workspace binding

RBAC establishes baseline access through groups and object-level privileges. ABAC adds policies that match governed tags and can apply across a catalog or schema. New assets can receive the relevant protection when they are tagged, reducing the need to configure every table independently.

Table-level row filters and column masks remain useful when the logic is specific to one table. Neither ABAC nor a table-level filter grants access by itself; both apply additional restrictions after the required object privileges have been granted.

For example, a table-specific column mask can return an SSN only to an approved account group:

CREATE FUNCTION finance.ssn_mask(ssn STRING)
RETURN CASE
  WHEN is_account_group_member('HumanResourceDept') THEN ssn ELSE '***-**-****'
END;
 
ALTER TABLE prod.finance.employees
ALTER COLUMN ssn
SET MASK finance.ssn_mask;

This pattern should be tested on the compute modes and Databricks Runtime versions used by the organization. Filters and masks also introduce operational and performance considerations, especially when their UDFs contain complex expressions or mapping-table lookups.

Govern External Storage Without Creating a Bypass

Unity Catalog can govern both managed and external data assets.

With managed assets, Unity Catalog manages the underlying storage lifecycle. With external assets, the data remains in a cloud location managed outside Databricks while Unity Catalog controls access from within the platform.

Two objects are central to external storage:

• Storage credential: Represents the cloud identity used to access storage.
• External location: Combines that credential with an approved cloud-storage path.

The control becomes ineffective if users can bypass Unity Catalog and access the underlying S3 bucket, ADLS container or GCS location directly. Direct cloud access should therefore be restricted, and privileges such as creating external locations should remain limited to a small platform-administration group.

Cloud IAM and Unity Catalog permissions must be designed together. Unity Catalog can govern platform access, but it cannot prevent a separate cloud identity from reading the same storage if that identity still has direct permission.

Improve Discovery with Governed Tags and Classification

Governance is not limited to blocking access. Users also need to find trustworthy data and understand what it represents.

Catalog Explorer helps users search and inspect governed assets. Descriptions, owners, tags, lineage and certification status can help distinguish approved data products from deprecated or experimental tables.

Governed tags are especially important when ABAC is used. A policy can target tables or columns carrying classifications such as:

• sensitivity = confidential
• data_type = pii
• domain = finance
• region = us

This separates two responsibilities. Data stewards classify assets, while platform or security teams define the policies applied to those classifications. The arrangement is more scalable than asking each table owner to recreate sensitive-data logic.

Classification still requires ownership and review. Automated detection can identify likely sensitive fields, but teams should validate the result and define what each classification means inside their organization.

Understand What Unity Catalog Lineage Captures

Unity Catalog can capture lineage across tables, notebooks, jobs, pipelines and dashboards, including column-level relationships where they can be inferred. This supports several practical tasks:

• Checking downstream impact before changing a schema
• Tracing an incorrect dashboard value to its source
• Understanding which pipelines read a sensitive table
• Helping new team members discover how an asset was produced
• Supporting audit and compliance investigations

However, lineage coverage is not universal. Current Databricks lineage documentation lists limitations involving RDDs, global temporary views, checkpointing, renamed objects and some path-based references. External lineage also requires additional configuration.

The current system-table paths are:

system.access.table_lineage
system.access.column_lineage

These tables contain a rolling one-year window. Catalog Explorer and the lineage API can retain supported lineage captured after September 1, 2024 for longer periods.

Teams should test representative notebooks, jobs, pipelines and external tools before depending on lineage for impact analysis. A lineage graph should be treated as captured evidence, not assumed evidence.

Query Audit Events Carefully

The audit log system table is available at:

system.access.audit

At the time of this update, Databricks documents this table as being in Public Preview. Most audit logs are region-specific, and some request parameters are masked unless the user has the appropriate administrative access.

A basic query for recent Unity Catalog events could be:

SELECT
  event_time,
  user_identity.email AS user_email,
  service_name,
  action_name,
  request_params
FROM system.access.audit
WHERE event_date >= current_date() - INTERVAL 30 DAYS
  AND service_name = 'unityCatalog'
ORDER BY event_time DESC;

Filtering on event_date is preferable for performance because the table is partitioned around that field. Before building compliance or security reporting on top of it, teams should confirm that the relevant system schema is enabled, authorized users can access it and the required event types are present.

Audit records may also need to feed a wider security or SIEM workflow. Unity Catalog provides queryable platform evidence, but incident alerting, retention policies and response procedures remain organizational responsibilities.

How Unity Catalog Supports Compliance

Unity Catalog can support a compliance control environment by helping organizations implement:

Control requirement	Relevant capability
Restrict access to sensitive records	Groups, privileges, row filters and column masks
Identify sensitive assets	Classification and governed tags
Trace data movement	Table-level and column-level lineage
Review platform activity	Audit logs and system tables
Separate production access	Catalog boundaries and workspace bindings
Share governed data	Unity Catalog sharing controls

These capabilities can contribute to GDPR, HIPAA, PCI DSS or internal security requirements when they are configured correctly. They do not make an organization compliant by themselves.

Compliance also depends on legal scope, lawful processing, retention, consent, encryption, cloud configuration, network security, incident response and documented operating procedures. Claims about regulatory readiness should therefore be reviewed by the organization's security, privacy and legal stakeholders.

Migrate from the Hive Metastore in Stages

A Unity Catalog migration should begin with discovery rather than immediate table movement.

1. Inventory: Document tables, storage paths, users, groups, service principals, grants and dependent workloads.
2. Identity: Move access management toward account-level groups and workload identities.
3. Namespace: Decide how catalogs and schemas represent environments and business ownership.
4. Storage: Configure managed storage, storage credentials and external locations.
5. Policies: Rebuild baseline grants and identify where ABAC, filters or masks are required.
6. Pilot: Migrate representative workloads before moving business-critical pipelines.
7. Validate: Test access, denial, lineage, auditing and downstream integrations.
8. Monitor: Review permission drift and retire redundant legacy access paths.

Legacy workloads may rely on unsupported compute modes, direct paths, workspace-local groups or libraries that behave differently under Unity Catalog. Identifying these dependencies during inventory is safer than discovering them during production migration.

Production Validation Checklist

Before rollout, confirm that:

• Account groups and service principals are synchronized correctly.
• Individual-user grants have been removed or justified.
• Catalog and schema traversal privileges behave as expected.
• Unauthorized users cannot read protected rows or raw column values.
• ABAC policies apply to newly tagged test assets.
• Service principals can run their required jobs without broader access.
• Direct access to governed storage paths is restricted.
• Representative workflows appear in lineage.
• Known lineage limitations are documented.
• Audit system tables are enabled and accessible.
• Break-glass access is limited, approved and logged.
• Catalog, policy and data owners are named.
• Permission and policy reviews have a defined schedule.

Common Unity Catalog Implementation Mistakes

The most common problems begin with architecture and ownership rather than SQL syntax.

• Granting privileges directly to individuals: This creates an access model that becomes difficult to audit and revoke.
• Using workspace-local identities: Unity Catalog governance should be based on account-level identities.
• Designing catalogs only around technical layers: This can obscure business ownership and production boundaries.
• Allowing direct storage access: Cloud permissions can bypass controls applied inside Unity Catalog.
• Applying separate masks to every table: ABAC may be more manageable when one rule must cover many tagged assets.
• Assuming lineage captures everything: Unsupported workflows can leave important gaps.
• Treating audit access as automatic: System tables, permissions, regional coverage and event availability must be verified.
• Migrating every workload together: A phased rollout makes incompatible workloads easier to identify and correct.
• Treating governance as a compliance certificate: Technology controls are only one part of compliance.
• Leaving policies without owners: Governance degrades when no team is responsible for reviewing classifications and access.

How Unity Catalog Connects to the Databricks Platform

Unity Catalog provides governance context across the wider Databricks platform.

Lakeflow workloads can operate on governed tables and contribute supported lineage records. Delta Lake provides storage reliability beneath many governed data workloads. Data quality monitoring helps teams profile data and identify anomalies. Registered models and other AI assets can be governed as securable objects.

Delta Sharing supports governed distribution beyond the organization. Revoking a share can prevent future authorized access, but organizations must still account for data that recipients may already have exported or retained.

For AI runtime interactions, Unity Gateway extends governance into areas such as model endpoints, agents and AI traffic. This distinction matters because governing the underlying table does not cover every risk created when an AI application uses that data.

Build Governance as an Operating Model

Unity Catalog centralizes enforcement and visibility, but successful governance also requires identity design, ownership, classification standards, validation and recurring access reviews. The strongest implementation is not the one with the most policies. It is the one in which every important boundary has an owner, a reason and a repeatable test.

For enterprises in the USA and other regulated markets, Unity Catalog controls should be connected to the organization's wider security, privacy and audit requirements rather than implemented as an isolated platform project.

Planning a Unity Catalog Implementation?

At Lucent Innovation, we help teams design Databricks governance foundations, migrate legacy workloads and validate access, lineage and operational controls before production rollout.

Explore our Databricks development services or review our capabilities as a Databricks Partner.