Access control
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Table-level access | GRANT/REVOKE via SQL, enforced at compute time — query is blocked before a row returnsWorks for notebooks, SQL warehouses, scheduled jobs, dashboards | Defines access policies but does not enforce inside DatabricksPolicies need UC or another tool to actually block access | UC winsGold tables are Delta in Databricks — only UC can block at query time |
| Row-level security | SQL row filters — regional manager sees only their region’s rows from the same table | Not supported at row level inside Databricks | UC winsCritical for multi-region or multi-department gold tables |
| Column masking (PII) | Mask Aadhaar, phone, email at query time — support analyst sees XXXX-1234 | Labels columns as sensitive; does not mask at query time in Databricks | UC winsThe lock on the door, not just the alarm |
| Auto sensitivity classification | Manual tagging only — no auto-detection of PII patterns | Auto-scans ADLS and UC; applies labels (PII, financial, confidential) across entire estate | Purview winsBetter at finding what needs protecting; but UC still does the protecting |
2
Data lineage
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Gold → bronze tracing | Automatic, real-time, column-level. Captured as code executes — never stale, zero configPython, SQL, Scala, R — all captured | Scan-based, periodic. Notebook-level lineage; SQL/job lineage still in previewAlways a lag between transformation and reflection | UC winsYour entire bronze → silver → gold path lives in Databricks |
| Cross-platform lineage (ADLS → Databricks → Power BI) | Stops at Databricks boundary — no downstream Power BI visibility | Traces lineage across ADLS, Databricks, Synapse, Power BI in one graph | Purview winsBut you debug upstream first — and upstream is all inside Databricks |
3
Data quality
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Gold table monitoring | Lakehouse Monitoring — continuous profiling on Delta: nulls, duplicates, distribution shifts, freshness, anomaly alertsQuality signals linked to lineage — trace spike to upstream cause | Rule-based quality scans, periodic. Less native to Delta formatData Observability view still in preview | UC winsContinuous monitoring on Delta is what Lakehouse Monitoring was built for |
| Who catches issues? | Your engineering team — they live in Databricks | Business-friendly quality dashboards | UC winsYour business users have low data knowledge — they’re not browsing quality dashboards. Engineers catch issues, and engineers live in Databricks |
4
Data discovery
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Engineer discovery | Catalog Explorer — fast, shows popularity, tags, freshness, ownership, lineage in one view | Broader search, less optimised for Databricks-native context | UC winsYour data team lives in Databricks — they’ll use Catalog Explorer daily |
| Business user discovery | Discover marketplace — curated data products, but still technical in feel | Unified Catalog with natural language search, data products, access request workflows, business glossaryDefine “Net Revenue” in plain language, link to exact column | Purview winsBut the critical question: are your business users actually browsing any catalog? With low data knowledge, they consume dashboards built by your team — they don’t search catalogs. Purview solves a problem you don’t have today. |
5
Audit and compliance
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Who accessed PII table? | Logs every query — user, timestamp, table, columns, rows returned. Compute-level granularity | Audit at scan/classification level, not query-level inside Databricks | UC winsQuery-level audit is what actually answers compliance questions |
| Compliance dashboards | Audit logs available; no built-in report templates | Built-in compliance dashboards; ties labels to policy reports | Purview winsBetter for presenting to leadership — but underlying evidence still comes from UC audit logs |
6
ADLS raw data governance
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Raw files in landing zone | No visibility — UC only sees registered Delta/Iceberg tables | Scans ADLS containers directly; catalogs every file, folder, and format | Purview winsReal gap — but raw ADLS is an engineering concern, not your primary governance need. Gold layer is what business users consume. |
7
Cost efficiency
| What matters | Unity Catalog | Purview | For your stack |
|---|---|---|---|
| Marginal cost | Included in Databricks Premium — zero additional costYou’re already paying for it | Pay-per-governed-asset/day. Scanning free since Jan 2025, but asset charges scale with estate size | UC winsFull governance for gold layer at no marginal cost |
Data Governance Analysis
Databricks Unity Catalog vs Microsoft Purview
Architecture: Azure ADLS + Databricks Gold Layer
1. Mental Model
Before comparing tools, the foundational distinction:
Unity Catalog — governs within Databricks for technical teams. Enforces access, captures lineage, monitors quality in real time inside the lakehouse.
Microsoft Purview — governs across your entire data estate for business users. Discovery, classification, compliance across Azure, M365, ADLS, multi-cloud.
They are complements, not competitors. Most enterprises running Databricks on Azure deploy both. The critical question is which to prioritise given your specific architecture.
2. Your Architecture
Understanding the stack before choosing a governance tool:
ADLS (raw storage) → Databricks Bronze → Silver → Gold → Business users consuming reports/dashboards
Key constraints that shape the recommendation:
- Data physically stored in Azure ADLS
- Gold layer built on Delta format inside Databricks
- Business users have low data knowledge — they consume dashboards, not browse catalogs
- Both Unity Catalog and Purview are already licensed
- Primary governance need: gold layer access, lineage, and quality
3. Governance Pillar Analysis
Pillar 1 — Access Control
The most important governance function. If someone should not see salary data or PII, the system must block them at query time — not flag it in a dashboard for someone to notice later.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Table-level access | GRANT/REVOKE via ANSI SQL, enforced at compute time. Query is blocked before a single row returns. Works for notebooks, SQL warehouses, scheduled jobs, and dashboards. | Defines access policies but does not enforce inside Databricks. Policies need UC or another tool to actually block access. | Use both |
| Row-level security | SQL row filters — regional manager sees only their region’s rows from the same gold table. | Not supported at row level inside Databricks. | Use both |
| Column masking (PII) | Mask Aadhaar, phone, email at query time. Support analyst sees XXXX-1234 instead of raw value. | Labels columns as sensitive but masking is not enforced at query time inside Databricks. | Use both |
| Sensitivity classification | Manual tagging only — no auto-detection of PII patterns across ADLS. | Auto-scans ADLS and UC metadata; applies sensitivity labels (PII, financial, confidential) across entire estate. | Use both |
| Estate-wide DLP policy | Scoped to Databricks only. | Enforces DLP across Azure, M365, ADLS, Power BI in one policy. | Use both |
Key insight: Unity Catalog is the lock on the door. Purview is the alarm system. The lock matters more — but Purview finds what needs locking.
Pillar 2 — Data Lineage
When a business user says ‘this revenue number looks wrong’, you need to trace the gold table back through silver transformations to bronze ingestion.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Gold → bronze tracing | Automatic, real-time, column-level. Captured as code executes — never stale, zero config. Works for Python, SQL, Scala, R. | Scan-based, periodic. Notebook-level lineage; SQL/job lineage still in preview. Always a lag between transformation and reflection. | Use both |
| Cross-platform lineage | Stops at Databricks boundary — no downstream Power BI visibility. | Traces lineage across ADLS, Databricks, Synapse, Power BI in one graph. | Use both |
| ADLS raw file lineage | No visibility — UC only sees registered Delta/Iceberg tables. | Scans ADLS containers; tracks raw files and folders as cataloged assets. | Use both |
Key insight: Your entire bronze → silver → gold path lives inside Databricks. UC gives you this automatically, in real time, at column level. You debug upstream first — and upstream is all inside Databricks.
Pillar 3 — Data Quality
Your business users have low data knowledge — they trust the number on the dashboard. If the gold table has nulls, duplicates, or stale data, they will make decisions on bad data without knowing it.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Gold table monitoring | Lakehouse Monitoring — continuous profiling on Delta: nulls, duplicates, distribution shifts, freshness, anomaly alerts. Quality signals linked to lineage — trace a spike back to the upstream cause. | Rule-based quality scans, periodic. Less native to Delta format. Data Observability view still in preview. | Use both |
| Quality visibility | Quality metrics in Catalog Explorer — built for technical audience. | Business-friendly quality dashboards; Data Observability view combines lineage and quality for non-technical users. | Use both |
| Who catches issues? | Your engineering team — they live in Databricks. UC’s monitoring alerts them directly where they work. | Better for presenting quality status to business stakeholders. | Use both |
Key insight: Continuous monitoring on Delta is what Lakehouse Monitoring was built for. Your engineers catch issues — and engineers live in Databricks.
Pillar 4 — Data Discovery
Critical question for your stack: are your business users actually going to browse any catalog? With low data knowledge, they typically consume dashboards built by your team — they do not search catalogs independently.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Engineer discovery | Catalog Explorer — fast, usage-oriented; shows popularity, tags, freshness, ownership, lineage in one view. | Broader search but less optimised for Databricks-native context. | Use both |
| Business user discovery | Discover marketplace (2025) — curated data products by domain. Still technical in feel. | Unified Catalog with natural language search, data products, access request workflows. Business-friendly UI. | Use both |
| Business glossary | Not supported — no place to define ‘Net Revenue’ or ‘Active Customer’ in plain language. | Full business glossary; link terms to columns; business users look up definitions without asking the data team. | Use both |
| Table & column descriptions | Add via SQL COMMENT ON — syncs to Purview automatically when scanned. Write once, both tools benefit. | Inherits descriptions from UC scan; business context layered on top. | Use both |
Key insight: Purview’s superior business discovery solves a problem you do not currently have. When business users start self-serving data, layer Purview then.
Pillar 5 — Audit and Compliance
If an auditor asks ‘who accessed the customer PII table in the last 90 days’, you need a clear, granular answer.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Query-level audit | Logs every query — user, timestamp, table, columns, rows returned. Compute-level granularity. This is what actually answers compliance questions. | Audit at scan/classification level, not query-level inside Databricks. Cannot answer ‘user X queried PII at 3:47pm Tuesday’. | Use both |
| Compliance dashboards | Audit logs available; no built-in compliance report templates for leadership. | Built-in compliance dashboards; ties sensitivity labels to policy reports across estate. | Use both |
| Access request workflow | No self-serve request flow — access granted by data team manually. | Data products have built-in access request and approval workflow for business users. | Use both |
Key insight: UC provides the evidence (query logs). Purview provides the presentation layer (dashboards). The evidence is more critical — but Purview’s compliance dashboards complement it well for reporting.
Pillar 6 — ADLS Raw Data Governance
Unity Catalog only sees Delta/Iceberg tables registered in its metastore. It has no visibility into raw files in your ADLS landing zone.
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Raw file cataloging | No visibility — UC only governs registered Delta tables. | Scans ADLS containers directly; catalogs every file, folder, and format. Applies sensitivity labels to raw data before Databricks ingests it. | Use both |
Key insight: Real capability gap for UC — but raw ADLS is an engineering concern, not the primary governance need. Your governance priority is the gold layer that business users consume.
Pillar 7 — Cost Efficiency
| Dimension | Unity Catalog | Microsoft Purview | Winner |
| Marginal cost | Included in Databricks Premium — zero additional cost. Full gold layer governance already paid for. | Pay-per-governed-asset per day. Scanning free since January 2025, but asset charges scale with estate size. | Use both |
4. Scorecard
| Pillar | UC | Purview | Notes |
| Access control (enforcement) | ✓ | UC enforces at compute time; Purview classifies only | |
| Data lineage (bronze → gold) | ✓ | Real-time, auto, column-level inside Databricks | |
| Data quality (gold tables) | ✓ | Lakehouse Monitoring on Delta, continuous | |
| Data discovery (engineers) | ✓ | Catalog Explorer — fast, usage-oriented | |
| Audit logs (query-level) | ✓ | Who accessed what, when, exactly | |
| Cost (marginal) | ✓ | Included in Premium; Purview charges per asset | |
| Sensitivity classification (auto) | ✓ | Purview auto-scans PII patterns across estate | |
| Cross-platform lineage | ✓ | ADLS → Databricks → Power BI in one graph | |
| Business glossary | ✓ | Plain language definitions for non-technical users | |
| ADLS raw file cataloging | ✓ | UC has no visibility of unregistered raw files | |
| Compliance dashboards | ✓ | Leadership reporting; UC provides underlying logs | |
| Access request workflow | ✓ | Self-serve for business users | |
| TOTAL | 6 | 6 | UC wins enforcement pillars. Purview wins visibility pillars. |
5. Practical Division of Work (Your Stack)
Given your architecture, here is exactly what each tool should own:
Unity Catalog — Primary Governance Engine (configure first)
- Grant/revoke table access on gold layer using SQL GRANT/REVOKE
- Set up row-level filters for multi-department or multi-region gold tables
- Column masking for PII fields (phone, Aadhaar, email)
- Enable Lakehouse Monitoring on all gold tables for continuous quality checks
- Add table and column comments — these sync into Purview automatically
- COMMENT ON TABLE gold.orders IS ‘Confirmed orders after payment validation. Source of truth for revenue reporting.’;
- COMMENT ON COLUMN gold.orders.net_revenue IS ‘Revenue after returns and discounts. Excludes GST.’;
- Review lineage graph to validate bronze → silver → gold transformation paths
Microsoft Purview — Compliance and Visibility Layer (add on top)
- Scan ADLS containers to catalog raw files in landing zone
- Set up sensitivity label auto-classification for PII across ADLS and UC assets
- Configure cross-platform lineage scan to connect Databricks assets to Power BI
- Build business glossary for key metrics your business users reference
- Create data products in Unified Catalog for self-serve access requests (when users are ready)
- Use compliance dashboards for regulatory reporting and leadership visibility
6. The One Action That Gives Double Value
Add COMMENT ON TABLE and COMMENT ON COLUMN on every gold table in Unity Catalog. Purview picks these up automatically during its next scan. Your business users see plain-language descriptions in Purview without any extra effort from your team.
7. Final Recommendation
Unity Catalog — Primary Governance Tool
Unity Catalog wins every pillar where governance is enforced — access control, lineage, quality monitoring, query audit, and cost. These are the pillars that actually prevent bad things from happening: unauthorised data access, untraced transformations, undetected quality issues.
Purview’s genuine strengths — business glossary, cross-platform catalog, auto-classification, compliance dashboards — matter in a world where business users actively self-serve data discovery. That is not your current reality. Your business users consume dashboards your team builds. Your critical data path is contained within Databricks. Your immediate governance pain is access control, lineage, and quality on the gold layer.
Recommended sequence:
- Now: Set up Unity Catalog — access controls, column masking, Lakehouse Monitoring, COMMENT ON on all gold tables
- Parallel: Use Purview for ADLS raw file scanning and sensitivity label auto-classification
- Later: Expand Purview when business users start self-serving data — business glossary, data products, access request flows
UC gives you operational governance of your gold layer at zero marginal cost. Purview adds the estate-wide visibility layer when your organisation is ready for it.
Leave a Reply