Data Governance Analysis
Databricks Unity Catalog vs Microsoft Purview
Architecture: Azure ADLS + Databricks Gold Layer
1. Mental Model
The foundational distinction before any tool comparison:
Unity Catalog — governs within Databricks for technical teams. Enforces access, captures lineage, monitors quality in real time inside the lakehouse.
Microsoft Purview — governs across your entire data estate for business users. Discovery, classification, compliance across Azure, M365, ADLS, multi-cloud.
They are complements, not competitors. The critical question is which to prioritise given your specific architecture and current business user maturity.
2. Your Architecture
ADLS (raw storage) → Databricks Bronze → Silver → Gold → Business users consuming reports/dashboards
Key constraints shaping the recommendation:
- Data physically stored in Azure ADLS
- Gold layer built on Delta format inside Databricks
- Business users have low data knowledge — they consume dashboards, they do not browse catalogs
- Both Unity Catalog and Purview are already licensed
- Primary governance need: gold layer access control, lineage, and quality
3. Governance Pillar Analysis
Pillar 1 — Access Control
The most important governance function. If someone should not see salary data or PII, the system must block them at query time — not flag it in a dashboard for someone to notice later. UC is the lock on the door; Purview is the alarm that finds what needs locking.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Table-level access | GRANT/REVOKE via ANSI SQL, enforced at compute time. Query blocked before a single row returns. Works for notebooks, SQL warehouses, jobs, dashboards. | Defines access policies but does not enforce inside Databricks. Policies need UC to actually block access. | UC wins |
| Row-level security | SQL row filters — regional manager sees only their region’s rows from the same gold table. | Not supported at row level inside Databricks. | UC wins |
| Column masking (PII) | Mask Aadhaar, phone, email at query time. Support analyst sees XXXX-1234 instead of raw value. | Labels columns as sensitive but masking is not enforced at query time inside Databricks. | UC wins |
| Sensitivity classification | Manual tagging only — no auto-detection of PII patterns across ADLS or UC metadata. | Auto-scans ADLS and UC; applies sensitivity labels (PII, financial, confidential) across entire estate automatically. | Purview wins |
| Estate-wide DLP policy | Scoped to Databricks only — cannot enforce DLP on ADLS raw files or Power BI. | Enforces DLP across Azure, M365, ADLS, Power BI in one unified policy. | Purview wins |
Key insight: UC enforces at compute time — that is what stops unauthorised access. Purview classifies what needs protecting. The lock matters more than the alarm, but both are needed.
Pillar 2 — Data Lineage
When a business user says ‘this revenue number looks wrong’, you need to trace the gold table back through silver transformations to bronze ingestion to find where the problem entered.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Gold → bronze tracing | Automatic, real-time, column-level. Captured as code executes — never stale, zero config. Covers Python, SQL, Scala, R notebooks. | Scan-based and periodic. Notebook-level lineage only; SQL/job/pipeline lineage still in preview. Always a lag after a transformation runs. | UC wins |
| Column-level lineage | Full column-to-column trace across all Databricks workloads — which input column fed which output column. | Column lineage available but limited; gaps for notebook code and non-scan sources. | UC wins |
| Cross-platform lineage | Stops at Databricks boundary — no downstream Power BI or upstream non-Databricks visibility. | Traces lineage across ADLS, Databricks, Synapse, Power BI in one end-to-end graph. | Purview wins |
| ADLS raw file lineage | No visibility — UC only sees registered Delta/Iceberg tables, not raw files. | Scans ADLS containers; tracks raw files and folders as cataloged assets with lineage. | Purview wins |
Key insight: Your entire bronze → silver → gold path lives inside Databricks. UC captures this automatically in real time. Cross-platform lineage matters but you debug upstream first — and upstream is all inside Databricks.
Pillar 3 — Data Quality
Your business users have low data knowledge — they trust the number on the dashboard. If the gold table has nulls, duplicates, or stale data, they will make decisions on bad data without knowing it.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Gold table monitoring | Lakehouse Monitoring — continuous profiling on Delta: nulls, duplicates, distribution shifts, freshness, anomaly alerts. Quality signals linked to lineage — trace a spike back to the upstream cause. | Rule-based quality scans, periodic. Less native to Delta format. Data Observability view still in preview. | UC wins |
| Anomaly detection | Automated — surfaces deviations from baseline directly in Catalog Explorer alongside governance metadata. | Scan-based detection; not continuous. Cannot match UC’s real-time depth on Delta tables. | UC wins |
| Quality visibility for business users | Quality metrics in Catalog Explorer — built for technical audience (data engineers). | Data Observability view (preview) combines lineage and quality in a business-friendly interface. | Purview wins |
Key insight: Your engineers catch issues, not business users. Continuous monitoring on Delta is what Lakehouse Monitoring was built for — and engineers live in Databricks, not Purview dashboards.
Pillar 4 — Data Discovery
Critical question for your stack: are your business users actually browsing any catalog? With low data knowledge, they typically consume dashboards built by your team — they do not search catalogs independently.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Engineer discovery | Catalog Explorer — fast, usage-oriented; shows popularity, tags, freshness, ownership, lineage in one view. Built for data engineers. | Broader search but less optimised for Databricks-native context and usage signals. | UC wins |
| Business user discovery | Discover marketplace (2025) — curated data products by domain. Still technical in feel. | Unified Catalog with natural language search, data products, access request workflows. Genuinely business-friendly UI. | Purview wins |
| Business glossary | Not supported — no way to define ‘Net Revenue’ or ‘Active Customer’ in plain language linked to a column. | Full business glossary; link terms to columns; business users look up definitions without asking the data team. | Purview wins |
| Table and column descriptionsWrite in UC → Purview inherits automatically | Add via SQL COMMENT ON — syncs to Purview automatically on next scan. Write once, both tools benefit. | Inherits descriptions from UC scan; business context and glossary links layered on top. | UC wins |
Key insight: Purview’s superior business discovery solves a problem you do not have today. Your business users are not self-serving data. When that changes, Purview is ready.
Pillar 5 — Audit and Compliance
When an auditor asks ‘who accessed the customer PII table in the last 90 days’, you need two things: the evidence (query logs) and the presentation (compliance report). UC provides the evidence; Purview provides the presentation. The evidence is what actually answers the compliance question.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Who accessed PII table? | Logs every query — user, timestamp, table, columns queried, rows returned. Compute-level granularity. This is the audit evidence. | Audit at scan/classification level only. Cannot answer ‘user X queried this PII column at 3:47pm Tuesday’ inside Databricks. | UC wins |
| Compliance dashboards | Audit logs available but no built-in compliance report templates for leadership or regulators. | Built-in compliance dashboards; ties sensitivity labels to policy reports across entire estate. | Purview wins |
| Access request workflow | No self-serve request flow — access granted manually by data team. | Data products have built-in access request and approval workflow; business users can self-serve. | Purview wins |
Key insight: UC provides the evidence that compliance questions require. Purview provides the presentation layer for leadership. Evidence is the primary need — but both have a role in a complete compliance posture.
Pillar 6 — ADLS Raw Data Governance
Unity Catalog governs Delta/Iceberg tables registered in its metastore. It has zero visibility into raw files sitting in ADLS containers — CSV dumps, JSON exports, Parquet files in a landing zone — before Databricks ingests them.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Raw file cataloging | No visibility — UC only governs registered Delta tables. Raw ADLS files are invisible to UC. | Scans ADLS containers directly; catalogs every file, folder, and format. Applies sensitivity labels to raw data before Databricks ingests it. | Purview wins |
Key insight: Real gap for UC — but raw ADLS is an engineering concern. Your governance priority is the gold layer that business users consume. Raw file governance is secondary.
Pillar 7 — Cost
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Marginal cost | Included in Databricks Premium — zero additional cost. Full gold layer governance is already paid for. | Pay-per-governed-asset per day. Scanning free since January 2025, but asset charges scale as your estate grows. | UC wins |
Key insight: UC gives you full gold layer governance at no marginal cost. Purview’s cost scales with how broadly you govern — estate-wide coverage comes at a price.
4. Scorecard — The Honest Tally
The scorecard below reflects a single winner per dimension — no ‘use both’ hedging. Where both tools contribute, the winner is the one doing the more critical job.
| Dimension | UC wins | Purview wins | Why |
| Table-level access enforcement | ✓ | UC enforces at compute time — Purview only classifies | |
| Row-level security | ✓ | UC only — Purview has no row-filter capability inside Databricks | |
| Column masking (PII) | ✓ | UC masks at query time — Purview labels but does not mask | |
| Gold → bronze lineage | ✓ | Real-time, auto, column-level — Purview is scan-based with lag | |
| Column-level lineage | ✓ | Full across all code types — Purview has gaps for notebooks | |
| Gold table quality monitoring | ✓ | Continuous on Delta — Purview is periodic and less native | |
| Engineer data discovery | ✓ | Catalog Explorer is faster and more usage-oriented | |
| Query-level audit (who accessed what) | ✓ | Compute-level logs — Purview only audits at scan/label level | |
| Cost | ✓ | Included in Premium — Purview charges per governed asset | |
| Sensitivity auto-classification | ✓ | Purview auto-scans PII patterns — UC requires manual tagging | |
| Cross-platform lineage (→ Power BI) | ✓ | Purview spans ADLS, Databricks, Synapse, Power BI | |
| ADLS raw file cataloging | ✓ | UC has no visibility of unregistered raw files | |
| Business glossary | ✓ | Purview only — UC has no plain-language term definitions | |
| Business user discovery | ✓ | Purview’s Unified Catalog is genuinely non-technical | |
| Compliance dashboards | ✓ | Purview has built-in policy reports — UC has raw logs only | |
| Access request workflow | ✓ | Purview data products have self-serve request flows | |
| TOTAL | 9 | 7 | UC wins all enforcement pillars. Purview wins all visibility pillars. Enforcement is the higher priority for your architecture. |
5. Practical Division of Work
Unity Catalog — Primary Governance Engine (configure first)
- Grant and revoke table access on gold layer using SQL GRANT/REVOKE statements
- Set up row-level filters for multi-department or multi-region gold tables
- Apply column masking for PII fields (phone, Aadhaar, email) at query time
- Enable Lakehouse Monitoring on all gold tables for continuous quality checks
- Add table and column COMMENT ON statements — these sync into Purview automatically
- COMMENT ON TABLE gold.orders IS ‘Confirmed orders after payment validation. Source of truth for revenue reporting.’;
- COMMENT ON COLUMN gold.orders.net_revenue IS ‘Revenue after returns and discounts. Excludes GST.’;
- Review lineage graph to validate bronze → silver → gold transformation paths
Microsoft Purview — Compliance and Visibility Layer (add on top)
- Scan ADLS containers to catalog raw files in the landing zone
- Configure sensitivity label auto-classification for PII across ADLS and UC assets
- Set up cross-platform lineage scan to connect Databricks assets to Power BI
- Build business glossary for key metrics your business users reference in reports
- Use compliance dashboards for regulatory reporting and leadership visibility
- Create data products with access request workflows when business users are ready to self-serve
6. The One Action That Gives Double Value
Add COMMENT ON TABLE and COMMENT ON COLUMN on every gold table in Unity Catalog. Purview picks these up automatically on its next scan. Your business users see plain-language descriptions in Purview without any extra effort from your team. One action, two tools benefit.
7. Final Recommendation
Unity Catalog — Primary Governance Tool
UC wins every pillar where governance is enforced — access control, lineage, quality monitoring, query audit, and cost. These are the pillars that actually prevent bad things from happening: unauthorised data access, untraced transformations, undetected quality issues, unanswered compliance questions.
Purview’s genuine strengths — business glossary, cross-platform catalog, auto-classification, compliance dashboards — matter when business users actively self-serve data discovery. That is not your current reality. Your business users consume dashboards your team builds. Your critical data path is contained within Databricks.
Recommended sequence:
- Now: Set up Unity Catalog — access controls, column masking, Lakehouse Monitoring, COMMENT ON on all gold tables
- Parallel: Use Purview for ADLS raw file scanning and sensitivity label auto-classification
- Later: Expand Purview when business users start self-serving data — business glossary, data products, access request flows
UC gives you operational governance of your gold layer at zero marginal cost. Purview adds the estate-wide visibility layer when your organisation is ready for it.
Leave a Reply