Capability Analysis

Unity Catalog vs Microsoft Purview — 13 Governance Capabilities

Architecture: Azure ADLS + Databricks Gold Layer  |  Includes UC-Only Impact Analysis

v4 — Updated with Data Lineage, AI/ML Governance, Data Sharing, Cost, Integration Details, and UC-Only Workarounds

Status legend:

  ✓ Best fit  — UC native & complete       ~ Partial  — UC has it with gaps       ✗ Cannot do  — Purview required

1. At-a-Glance Summary — All 13 Capabilities

Three new capabilities added in v4: Data Lineage, AI/ML Governance, and Data Sharing & Collaboration — all previously discussed but missing from the capability matrix.

Capability	UC Status	Unity Catalog	Microsoft Purview	Best Tool
Business Glossary	✗ Cannot do	COMMENT ON only — no governed terms	Full glossary: definitions, owners, stewards, column links	Purview wins
Taxonomy	✗ Cannot do	Flat catalog.schema.table namespace only	Hierarchical taxonomy; domain assignment; custom rules	Purview wins
Data Catalog	~ Partial	Catalog Explorer for Databricks assets only	Unified catalog: 200+ sources incl. ADLS, SQL, Power BI	Purview wins
Data Discovery	~ Partial	Engineer-friendly; no NL search or glossary	Natural language search; business-friendly; self-serve	Purview wins
Audit Log	✓ Best fit	Query-level: user, table, columns, rows, timestamp	Scan/classification level; no query audit inside Databricks	UC wins
Data Quality	✓ Best fit	Lakehouse Monitoring: continuous on Delta	Periodic rule-based scans; not native to Delta	UC wins
Access Control	✓ Best fit	Row, column, table — enforced at compute time	Policy definition only; enforcement requires UC	UC wins
Privacy	~ Partial	Column masking, row filters — Delta only	Sensitivity labels, DSPM, DLP across full estate	Purview wins
Security	~ Partial	Fine-grained inside Databricks only	Estate-wide DLP, DSPM for AI, Defender integration	Purview wins
Data Classification	✗ Cannot do	Manual tags only; no auto-scan	200+ auto-classifiers across ADLS, UC, Azure	Purview wins
Data Lineage	✓ Best fit	Real-time, auto, column-level in Databricks	Scan-based with lag; cross-platform (→ Power BI)	UC wins
AI/ML Governance	✓ Best fit	MLflow models, features, agents in UC	DSPM for Copilot/AI; no model lifecycle governance	UC wins
Data Sharing	✓ Best fit	Delta Sharing: open protocol, cross-platform	Data products with access request; no live sharing protocol	UC wins

Pattern: UC wins 6 (enforcement + ML + sharing). Purview wins 4 (metadata + classification). 3 are split (catalog, discovery, privacy/security). The new capabilities (lineage, AI/ML, sharing) all favour UC — reinforcing it as the primary tool for your Databricks-centric stack.

2. Capability Deep-Dive — All 13 Capabilities

Capability 1 — Business Glossary

Defines data terms in plain language and links them to physical columns. Bridges the gap between technical data and business understanding.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Term definitionBusiness language meaning	COMMENT ON TABLE/COLUMN only — technical description, no governed term with ownership or approval	Full glossary: rich-text definitions, synonyms, examples, related terms, domain assignment	✗ Cannot do
Term ownershipAccountability for definition	No concept of term owner separate from table owner	Each term has an assigned steward and approval workflow	✗ Cannot do
Term-to-column linkingConnect term to physical column	Not supported — COMMENTs are text, not linked governed terms	Glossary terms linked to columns; bi-directional navigation	✗ Cannot do
Cross-system glossarySame term across Databricks + Power BI	Scoped to Databricks only	Single glossary across 200+ connected sources	✗ Cannot do
Inheritance into PurviewUC COMMENTs flow to Purview	Yes — COMMENT ON syncs on next Purview scan. Write once, both benefit.	Inherits UC descriptions; glossary terms linked on top	~ Partial

Note: UC has no business glossary. Write COMMENT ON in UC — Purview inherits it automatically. All governed term management happens in Purview.

Capability 2 — Taxonomy

Hierarchical classification: domains, sub-domains, categories (e.g. Finance > Revenue > Net Revenue). Gives structure to how data is grouped and governed.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Hierarchical classificationNested categories/domains	No taxonomy — flat three-level namespace: catalog.schema.table	Full hierarchical taxonomy; assets classified within domain hierarchy	✗ Cannot do
Custom classification rulesAuto-assign taxonomy labels	Not supported	Custom rules based on column name, data patterns, or metadata	✗ Cannot do
Domain assignmentAssign table to Finance, HR etc.	Not native — schema naming convention workaround (gold.finance.x)	Assets formally assigned to domains; domain owners defined	✗ Cannot do
Flat taggingKey-value tags on assets	Yes — tags on catalogs, schemas, tables, columns. Useful but not hierarchical.	Yes — plus hierarchical classification on top	~ Partial

Note: UC has no taxonomy. Use schema naming conventions as workaround. Purview for formal domain hierarchy.

Capability 3 — Data Catalog

Searchable inventory of all data assets with metadata, descriptions, lineage, and ownership. Scope is the key differentiator. v4 adds Lakehouse Federation and external catalogs.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Databricks asset catalogTables, views, models, notebooks	Full — all registered Delta/Iceberg assets with lineage, ownership, tags, usage signals	Cataloged via scan; less native context	✓ Best fit
ADLS raw file catalogCSV, JSON, Parquet before ingestion	Not supported — UC only sees registered tables	Full — scans ADLS containers; catalogs files, folders, formats	✗ Cannot do
Cross-source catalogSQL Server, Power BI, Synapse, Salesforce	Not supported natively — Lakehouse Federation (v4 addition) lets UC query external sources like PostgreSQL, MySQL, SQL Server but does not catalog them as governed assets	200+ connectors; single catalog spanning all data sources	✗ Cannot do
Lakehouse FederationQuery external DBs without ingestion	Yes — UC can query external PostgreSQL, MySQL, SQL Server, Redshift, BigQuery without copying data. Read-only, no governance on the external source.	Not applicable — Purview catalogs external sources but does not query them	✓ Best fit
Usage / popularity signalsWhich tables are most queried?	Yes — query frequency tracked, hot assets surfaced in Catalog Explorer	Not available — no query-level usage signals in Purview	✓ Best fit
Asset ownershipWho owns this table?	Table/schema owners via GRANT or ownership transfer; clear accountability chain	Asset owners set in Purview; linked to business contacts	✓ Best fit

Note: UC is the best catalog for Databricks assets — it has usage signals, real-time metadata, and Lakehouse Federation for querying external DBs. Purview is the best catalog for everything outside Databricks. For your ADLS + Databricks stack, both have a role.

Capability 4 — Data Discovery

How users find the right data asset. Critical split: engineer discovery (technical, fast) vs business user discovery (non-technical, self-serve).

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Engineer discoveryFind tables to build pipelines	Catalog Explorer — usage signals, lineage, tags, freshness, ownership. Best in class for engineers.	Broader but lacks Databricks-native context and usage signals	✓ Best fit
Business user discoveryNon-technical users finding data	Discover marketplace (2025) — improving but still technical in feel	Unified Catalog: NL search, glossary links, data product cards, access request	~ Partial
Natural language searchSearch by question not table name	Not supported	Yes — ‘show me sales data for Q3’ returns relevant assets	✗ Cannot do
AI-powered recommendationsSurface relevant assets by role/history	AI-powered signals in Discover — certification, quality, usage patterns	AI recommendations based on user role and usage patterns	~ Partial
Saved searches / alertsNotify when new relevant asset published	Not supported	Subscribe to search alerts and asset change notifications	✗ Cannot do

Note: Your business users are not self-serving today. UC’s engineer discovery is the daily driver. When business users mature, Purview is ready.

Capability 5 — Audit Log

Who accessed what, when, and what did they do? v4 adds system tables detail — UC stores audit logs in queryable Delta system tables, not just flat logs.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Query-level auditWho ran SELECT on this table?	Full — system tables (system.access.audit) log every query: user, timestamp, table, columns, rows, notebook/job. Queryable with SQL.	Not available — Purview sees scan metadata, not live queries	✓ Best fit
Column-level auditWhich columns did user X access?	Yes — logs specific columns queried, not just the table	Not available at column level inside Databricks	✓ Best fit
Failed access attemptsWho tried and was blocked?	Yes — logs both successful and denied access attempts with reason	Not available inside Databricks	✓ Best fit
Table access historyHow often is this table queried?	Yes — system.access.table_lineage provides access patterns over time	Not available — Purview has no query-frequency data	✓ Best fit
Cross-platform auditAudit across Azure, Power BI, M365	Not available — UC audit is Databricks-only	Unified audit log across all Azure services, Power BI, M365	✗ Cannot do
Compliance reportingPre-built reports for GDPR, audit	No built-in templates — you query system tables and build reports in SQL or notebooks	Built-in compliance dashboards; sensitivity label reports; policy violation reports	✗ Cannot do
Log retention & exportHow long? Where?	Delta system tables in your ADLS — indefinite retention, queryable, exportable to SIEM	Azure Monitor; configurable retention; SIEM integration via Sentinel	✓ Best fit

Note: v4 clarification: UC audit lives in Delta system tables (system.access.audit), not just log files. This means you can query audit history with SQL, join it with lineage tables, and build custom compliance views — powerful but requires engineering effort.

Capability 6 — Data Quality Measurement

Detects problems before they reach business users. v4 adds Delta Live Tables (DLT) expectations — declarative quality constraints enforced at write time, not just monitored.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Continuous profilingMonitor gold tables around the clock	Lakehouse Monitoring: continuously profiles Delta — nulls, distinct counts, distribution histograms. Runs on Delta log, not full scans.	Periodic scan-based profiling. Not native to Delta.	✓ Best fit
DLT ExpectationsDeclarative quality rules enforced at write	Delta Live Tables expectations: EXPECT (warn), EXPECT OR DROP (filter), EXPECT OR FAIL (halt). Quality enforced at pipeline level before data reaches gold.	No equivalent — Purview cannot enforce quality at write time	✓ Best fit
Freshness monitoringAlert when table stops updating	Yes — tracks modification timestamps; alerts on staleness	Limited — only detectable via scan schedule lag	✓ Best fit
Anomaly detectionAlert on baseline deviation	Yes — statistical baselines; alerts on distribution shifts	Rule-based thresholds only; no statistical anomaly detection	✓ Best fit
Quality-to-lineage linkTrace quality issue to upstream source	Yes — quality alerts link to lineage graph; engineers see upstream cause	No direct link — quality and lineage are separate views	✓ Best fit
Table constraintsNOT NULL, CHECK at write time	Yes — Delta table constraints enforced at write: NOT NULL, CHECK constraints prevent bad data from entering	Not applicable — Purview does not enforce at write time	✓ Best fit
Business-friendly quality viewNon-technical quality dashboard	Catalog Explorer — technical audience	Data Observability view (preview) — business-friendly quality + lineage	✗ Cannot do

Note: v4 addition: DLT Expectations are UC’s secret weapon for quality — they enforce rules at write time (before data enters gold) rather than just detecting problems after they arrive. Combined with Lakehouse Monitoring for continuous observation and Delta constraints for schema enforcement, UC has the most complete quality stack for Delta tables.

Capability 7 — Access Control

Who can see what data. v4 adds credential passthrough and external locations — important for your ADLS storage layer.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Table-level accessGrant/revoke SELECT	GRANT/REVOKE via SQL — enforced at compute time, every query, no bypass	Defines policies; does not enforce inside Databricks	✓ Best fit
Row-level securityDifferent users see different rows	Row filters via SQL — regional manager sees only their region	Not supported for row-level inside Databricks	✓ Best fit
Column maskingHide PII columns	Column masks via SQL — phone returns XXXX-1234 for unauthorised users	Sensitivity labels identify PII; masking requires UC	✓ Best fit
ABACAccess based on user attributes	Yes — policies based on department, clearance, custom attributes	Policy definition only; enforcement needs UC	✓ Best fit
Credential passthroughUser identity flows to ADLS	Yes — UC can pass Azure AD identity to ADLS, ensuring storage-level access matches UC grants. Critical for your ADLS-backed tables.	Not applicable — Purview does not manage compute identity	✓ Best fit
External locationsGovern ADLS paths as UC objects	Yes — register ADLS paths as external locations in UC; apply access control to storage paths, not just tables	Not applicable — different governance model	✓ Best fit
Access request workflowSelf-serve access requests	No self-serve flow — manual SQL GRANT by data team	Data products with built-in access request and approval workflow	✗ Cannot do
ADLS raw file accessControl access to raw landing zone	Not applicable for unregistered files — but external locations can govern specific ADLS paths	Purview can define policies for ADLS at file/folder level	~ Partial

Note: v4 additions: credential passthrough and external locations are critical for your ADLS + Databricks architecture. Credential passthrough ensures Azure AD identity flows to storage. External locations let you register and govern specific ADLS paths as UC objects — bridging the gap between table-level and storage-level governance.

Capability 8 — Privacy

Protecting PII from unauthorised access and managing data subject rights. UC handles masking for Delta; Purview handles estate-wide discovery and GDPR workflows.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
PII column maskingMask Aadhaar, phone at query time	Column masking for registered Delta tables. Analyst sees XXXX-1234.	Identifies PII; does not mask inside Databricks	✓ Best fit
PII auto-detectionFind PII automatically	Not supported — manual tagging only	200+ classifiers auto-detect PII patterns across ADLS and UC	✗ Cannot do
Right to erasureGDPR Article 17	Not supported — manual DELETE per table	Data map identifies subject’s data; supports erasure workflow	✗ Cannot do
Data minimisationAlert on unnecessary PII	Not supported	Sensitivity labels + DSPM identify overexposed PII	✗ Cannot do
DSPM for AIAI agents accessing sensitive data	Not applicable	Flags which Copilot/AI agents access sensitive data; remediation workflow	✗ Cannot do
Privacy scopeWhich data is covered?	Only registered Delta tables	Full estate: ADLS, Delta, SQL, Power BI, M365	~ Partial

Note: Privacy is split. UC enforces protection inside Databricks. Purview discovers where PII exists estate-wide. You need both for a complete privacy posture.

Capability 9 — Security

Identity, network controls, threat detection, and data loss prevention. UC provides deep security inside Databricks; Purview manages posture across the Microsoft estate.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Identity & accessAzure AD / Entra ID integration	Full integration; service principals; managed identities supported	Full Azure AD; conditional access; Privileged Identity Management	✓ Best fit
Fine-grained permissionsGranular control below table level	Row filters, column masks, ABAC — most granular in Databricks	Policy-level; granular enforcement requires UC	✓ Best fit
DLPPrevent sensitive data exfiltration	Not supported — UC enforces access but no DLP for export/sharing	DLP across Azure, M365, Power BI — alert and block	✗ Cannot do
Threat detectionUnusual access patterns	Not built-in — logs go to Sentinel externally	Microsoft Defender integration; anomalous patterns flagged	✗ Cannot do
Network controlsRestrict by network/IP	Workspace-level: private link, VNet injection (not UC-specific)	Purview has its own network controls; complements Databricks	~ Partial
Security audit trailEvidence for investigations	Compute-level query logs — every access inside Databricks	Cross-platform audit — Azure Activity Log, M365 unified	✓ Best fit

Note: UC secures inside Databricks with finest granularity. Purview secures the perimeter — DLP, threat detection, DSPM. One does not replace the other.

Capability 10 — Data Classification

Automatic identification and labeling by sensitivity and type. Purview’s strongest exclusive capability — UC cannot auto-classify.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Auto-classificationScan and auto-apply labels	Not supported — no scanner reads values/names to classify	200+ classifiers; scans column names, data samples, metadata; auto-labels at scale	✗ Cannot do
Custom classifiersDomain-specific classification	Not supported	Custom regex, keyword, or ML-based classifiers	✗ Cannot do
Manual taggingApply tags manually	Yes — tags on catalogs, schemas, tables, columns. Manual only.	Yes — plus auto-classification on top	~ Partial
Label propagationLabel flows downstream	Not supported	Sensitivity labels propagate to derived columns automatically	✗ Cannot do
Classification scopeWhich assets?	Registered Delta tables only, manual	ADLS, Delta, SQL, Power BI, CosmosDB — full estate	✗ Cannot do
Classification → accessLabels trigger access policy?	Not automatic — engineers manually configure masking after tagging	Sensitivity labels trigger DLP and conditional access automatically	✗ Cannot do

Note: Classification is where UC genuinely cannot help. No scanner, no auto-detect. Once Purview classifies a column as PII, engineers use that signal to set up masking in UC. This is the strongest argument for adding minimum Purview configuration even in a UC-only strategy.

Capability 11 — Data Lineage  [NEW in v4]

Traces how data flows from source to consumption — bronze → silver → gold → dashboard. Critical for debugging (‘this number looks wrong’), impact analysis (‘if I change this column, what breaks?’), and compliance (‘prove the data trail’). This was covered in the earlier governance document but was missing from the capability matrix.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Bronze → gold lineageTrace gold table back to raw source	Automatic, real-time, column-level. Captured as code executes in notebooks and jobs — zero config, never stale. Works for Python, SQL, Scala, R.	Scan-based, periodic. Notebook-level lineage; SQL/job lineage still in preview. Always a lag.	✓ Best fit
Column-level lineageWhich input column fed which output column?	Full column-to-column trace across all Databricks workloads — automatic, no annotation required	Column lineage available but limited; gaps for notebook code	✓ Best fit
Pipeline lineageDLT and job lineage	Yes — Delta Live Tables pipelines, Databricks Workflows, and Spark jobs all captured in UC lineage graph automatically	Not captured at pipeline level; only entity-level from scans	✓ Best fit
Cross-platform lineageADLS → Databricks → Power BI	Stops at Databricks boundary — no downstream Power BI or upstream non-Databricks visibility	Traces across ADLS, Databricks, Synapse, Power BI in one end-to-end graph	✗ Cannot do
Impact analysisIf I change column X, what breaks?	Yes — downstream dependents visible in lineage graph; before modifying a gold column, see which dashboards and notebooks depend on it	Available across Purview-connected sources; broader scope	✓ Best fit
Lineage storageWhere is lineage data stored?	System tables: system.access.table_lineage and system.access.column_lineage — queryable with SQL, joinable with audit tables	Purview Data Map — visual graph in portal, exportable via API	✓ Best fit
ADLS raw file lineageTrack raw files before ingestion	Not tracked — UC only sees registered Delta/Iceberg tables	Scans ADLS containers; tracks files and folders as assets in lineage	✗ Cannot do

Note: Lineage is UC’s second strongest capability after access control. Your entire bronze → silver → gold path lives inside Databricks. UC captures this automatically, in real time, at column level, stored in queryable system tables. The only gap: cross-platform lineage beyond Databricks — Purview fills this.

Capability 12 — AI/ML Governance  [NEW in v4]

Governing the ML lifecycle: model registration, experiment tracking, feature store, model serving, and AI agent governance. Increasingly critical as organisations deploy models and agents on top of their data. This was mentioned in earlier analysis but missing from the capability matrix.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Model registryRegister, version, and stage ML models	UC Models — central registry for MLflow models. Version tracking, stage transitions (staging → production), approval gates.	Not applicable — Purview does not manage ML model lifecycle	✓ Best fit
Model lineageTraining data → model → endpoint	Full: which training data, which features, which experiment run, which serving endpoint. End-to-end traced in UC.	Not available — Purview has no model lifecycle visibility	✓ Best fit
Feature storeGoverned, reusable ML features	Databricks Feature Store — features registered in UC as tables with lineage to training datasets and models	Not applicable	✓ Best fit
Experiment trackingCompare model runs and metrics	MLflow experiments — all logged in Databricks; metrics, parameters, artifacts tracked per run	Not applicable	✓ Best fit
AI agent governanceGovern tools and data AI agents access	UC governs which data and tools a Databricks AI agent can access — permissions enforced at runtime	DSPM for AI — flags which data Copilot and Azure AI agents access; remediation workflow. Different scope.	~ Partial
Model serving auditWho called which model endpoint?	Yes — serving endpoints logged; inference tables capture inputs and outputs for monitoring	Not available inside Databricks	✓ Best fit
Copilot governanceControl Microsoft Copilot data access	Not applicable — UC does not govern M365 Copilot	DSPM flags overexposed data accessible by Copilot; sensitivity labels restrict access	✗ Cannot do

Note: AI/ML governance is almost entirely UC’s domain. If you are running ML models on Databricks (MLflow, Feature Store, Model Serving), UC is the only tool that governs the full lifecycle. The one area Purview adds: governing data access for Microsoft Copilot and Azure AI agents outside Databricks.

Capability 13 — Data Sharing & Collaboration  [NEW in v4]

Sharing data across teams, organisations, and platforms. Delta Sharing — Databricks’ open protocol for secure cross-platform data sharing — is a key UC capability never covered in previous versions.

Dimension / Sub-point	Unity Catalog (UC)	Microsoft Purview	UC Status
Delta SharingOpen protocol for live data sharing	Yes — share live Delta tables without copying data. Recipients can be on any platform (Spark, pandas, Power BI, Tableau). Open protocol under Linux Foundation.	Not applicable — Purview does not have a live data sharing protocol	✓ Best fit
Internal sharing (across workspaces)Share data across Databricks workspaces	Yes — UC metastore spans workspaces. Grant access to tables across workspaces without data duplication.	Not applicable — different governance model	✓ Best fit
External sharing (cross-org)Share data with partners or vendors	Delta Sharing recipients — external organisations can access specific tables/partitions with audit trail. No data leaves your ADLS unless explicitly shared.	Not applicable — Purview does not manage external data sharing	✓ Best fit
MarketplacePublish/consume shared datasets	Databricks Marketplace — publish and discover datasets, notebooks, ML models. Both free and paid listings.	Not applicable	✓ Best fit
Data products (packaged)Curated bundles with access workflow	Discover marketplace — data products with certification, domain tagging. Still evolving.	Data products with owner, description, access request flow, SLA — more mature business packaging	~ Partial
Share auditWho accessed shared data?	Yes — Delta Sharing audit logs capture recipient access: who, when, which table, how many rows	Not applicable	✓ Best fit

Note: Data Sharing is exclusively UC territory. Delta Sharing is the only open protocol for live, governed cross-platform data sharing without copying. If you share data externally (partners, vendors, other platforms), UC is the only option. Purview’s data products complement this for internal business user discovery and access requests.

3. Cost Comparison  [NEW in v4]

Previously missing from the capability analysis. Critical for a UC-only vs both-tools decision.

Cost Dimension	Unity Catalog	Microsoft Purview	Winner
Base licensing	Included in Databricks Premium — zero additional cost for full governance	Pay-per-governed-asset per day. Scanning free since Jan 2025. Asset charges scale with estate size.	UC wins
Scaling cost	Flat — no per-table or per-asset charge as your gold layer grows	Linear — cost grows with number of governed assets (tables, files, columns)	UC wins
Engineering effort	Low for governance; high if you need to build compliance reports manually from audit system tables	Low for compliance reporting (built-in dashboards); higher initial scan configuration	Purview wins
Total cost of ownership	Zero marginal for gold layer governance. Compliance reporting requires engineering time.	Predictable per-asset pricing. Compliance dashboards are turnkey but at a cost.	UC wins

Bottom line: UC gives you gold layer governance at zero marginal cost. Purview’s cost is proportional to how broadly you govern. If you already have both licenses, the question is not cost — it is configuration effort vs compliance risk.

4. Technical Integration Details  [NEW in v4]

How Purview actually scans UC, what metadata flows automatically, and what does not. Previously missing — critical for planning.

What Flows	Direction	What Syncs	What Does Not Sync
Table metadata	UC → Purview (scan)	Table names, schemas, column names, data types, COMMENT ON descriptions	Row-level data, query patterns, usage frequency, popularity signals
Column descriptions	UC → Purview (scan)	COMMENT ON COLUMN text syncs automatically on scan	Custom tags and UC-specific metadata may not all map to Purview schema
Lineage	UC → Purview (partial)	Entity-level lineage (which notebook wrote to which table)	Column-level lineage, pipeline-level lineage, DLT lineage — gaps in current integration
Sensitivity labels	Purview → UC (manual)	Engineers see Purview labels as signal to configure UC masking	Labels do not auto-trigger UC access policies — manual implementation required
Access policies	No sync		UC access controls and Purview policies are independent — no bi-directional sync
Business glossary	No sync		Purview glossary terms are not pushed into UC — they exist only in Purview
Audit logs	No sync		UC audit (system tables) and Purview audit (Azure Monitor) are separate systems

Integration gaps: Scoped and incremental UC scans are not yet supported in Purview — every scan is full. Column-level lineage sync is limited. Sensitivity labels from Purview do not auto-trigger UC access policies. Plan for manual bridging between the two systems.

Practical implication: The integration is metadata-level only. Purview reads UC’s catalog; UC never reads Purview. Access enforcement, lineage depth, and quality monitoring remain independent in UC regardless of Purview configuration.

5. UC-Only Analysis — What Happens If You Drop Purview

v4 update: now includes a Workaround column for every capability you lose, so you can evaluate UC-only with clear fallback options.

5a — What You Lose Completely (with Workarounds)

These capabilities have zero equivalent in UC. Workarounds are manual processes, not tool equivalents.

Capability	What You Lose	Impact	Workaround
Business Glossary	No governed terms with ownership, approval, column links	Business users keep asking data team what columns mean	Use Confluence/Notion wiki page for term definitions; link to UC table names manually
Taxonomy	No domain hierarchy above catalog.schema.table	No formal domain governance	Use schema naming: gold.finance.*, gold.hr.*, gold.sales.* as manual domain structure
Auto Classification	No PII auto-scan; every column must be manually tagged	PII columns get missed; compliance exposure	Build a PII audit notebook: regex scan column names + sample values for common patterns (email, phone, Aadhaar). Run monthly.
ADLS raw file catalog	Raw landing zone completely invisible	Ungoverned blind spot pre-ingestion	Use Azure Storage Explorer + naming conventions; document landing zone structure in wiki
DLP policies	Nothing prevents sensitive data export	GDPR/internal policy compliance gap	Restrict DBSQL result download permissions; disable notebook result export for sensitive workspaces
DSPM for AI	No visibility of AI agent data access	Relevant if using Copilot/Azure AI	Manually audit AI agent configurations; restrict service principal permissions in UC
Right to erasure	No estate-wide subject data location	Manual hunt through tables per request	Build a ‘PII map’ notebook that queries all gold tables for a subject’s data by identifier column
Cross-platform lineage	No downstream Power BI visibility	Cannot trace end-to-end data journey	Document Power BI dataset-to-gold-table mapping manually; maintain in wiki or spreadsheet
Compliance dashboards	No pre-built audit reports	Build from scratch in SQL	Create notebooks querying system.access.audit; schedule as dashboard in Databricks SQL
Access request workflow	No self-serve for business users	Data team bottleneck for every grant	Use Jira/ServiceNow ticket workflow: user requests → approver reviews → engineer runs GRANT
Natural language search	No NL discovery for business users	Business users cannot find data themselves	Build a ‘data catalog’ page in wiki/Confluence with table descriptions searchable by keyword

5b — What You Get Partially

Capability	What UC Does	What You Miss
Data Catalog	Full catalog for Databricks assets; Lakehouse Federation for querying external DBs	ADLS raw files, SQL Server, Power BI assets are not cataloged
Data Discovery	Catalog Explorer excellent for engineers; Discover marketplace improving	Business users have no self-serve discovery tool
Privacy	Column masking, row filters on Delta tables	Must manually identify PII first — no auto-detection at scale
Security	Fine-grained access + credential passthrough + external locations	No DLP, no threat detection across broader estate
Data Classification	Manual tags on all UC objects	No auto-scan means only manually reviewed columns are classified

5c — The Four Real Risks of UC-Only

Risk 1 — PII Blindspots

Without auto-classification, you only protect PII columns you know about. Every new table is unclassified until an engineer manually reviews it. One missed Aadhaar or phone column is a regulatory risk. The PII audit notebook workaround helps but requires discipline — if the monthly scan is skipped, new PII slips through.

Risk 2 — No Compliance Reporting

When a regulator or auditor asks for a sensitivity report, you build everything from UC’s system.access.audit tables in SQL. This is feasible but expensive in engineering time — expect 2-3 days per compliance report, recurring quarterly. Purview’s built-in dashboards eliminate this effort entirely.

Risk 3 — Business Users Stay Dependent

Every question about ‘what data exists’, ‘what does this column mean’, and ‘can I get access’ lands on your data team. The Confluence/wiki workaround helps but goes stale quickly and has no linked navigation to actual tables. This bottleneck grows linearly with your user base.

Risk 4 — ADLS Blind Spot

Your raw landing zone in ADLS is completely ungoverned. If sensitive data lands there and is delayed in ingestion — or never ingested — it sits invisible with no access audit, no sensitivity labels, and no cataloging. This is your largest uncontrolled data surface and your biggest compliance gap.

5d — Where UC-Only Is Genuinely Sufficient

If your scope is ‘secure and monitor the gold layer for my technical team’, these capabilities are fully covered:

Capability	What UC Covers	Confidence
Gold table access control	GRANT/REVOKE, row filters, column masks, ABAC, credential passthrough	Complete ✓
Bronze → gold lineage	Automatic, real-time, column-level — zero config	Complete ✓
Data quality monitoring	Lakehouse Monitoring + DLT Expectations + Delta constraints	Complete ✓
Query-level audit	System tables: user, timestamp, table, columns, rows — indefinite retention	Complete ✓
Column masking (known PII)	Masks configured columns — powerful but requires manual PII identification	Sufficient ✓
Engineer data discovery	Catalog Explorer — best in class	Complete ✓
ML model governance	MLflow models, features, experiments, serving — full lifecycle in UC	Complete ✓
Data sharing	Delta Sharing — open protocol, cross-platform, audited	Complete ✓
External DB federation	Lakehouse Federation — query PostgreSQL, MySQL, SQL Server without copying data	Complete ✓

6. What UC Cannot Do — Consolidated Reference

All gaps and partials in one table. Updated in v4 with 3 additional capabilities.

Capability	Status	What UC Cannot Do	Use Purview For
Business Glossary	✗ Cannot	Define governed business terms with ownership and column linking	Purview business glossary
Taxonomy	✗ Cannot	Organise data into domain hierarchy above namespace	Purview hierarchical taxonomy
Data Classification	✗ Cannot	Auto-scan to detect PII, financial, or sensitive patterns	Purview 200+ auto-classifiers
Cross-source catalog	✗ Cannot	Catalog ADLS raw files, SQL Server, Power BI	Purview 200+ source catalog
ADLS raw file governance	✗ Cannot	See or govern raw files before Delta ingestion	Purview ADLS scanner
Natural language discovery	✗ Cannot	Let business users search in plain language	Purview Unified Catalog NL search
Right to erasure (GDPR)	✗ Cannot	Locate + delete subject data across estate	Purview data subject workflow
DLP policies	✗ Cannot	Prevent sensitive data export/sharing	Purview DLP across Azure, M365
DSPM for AI	✗ Cannot	Detect AI agent access to sensitive data	Purview DSPM for AI
Cross-platform audit	✗ Cannot	Unified audit across Azure, M365, Power BI	Purview + Azure Monitor
Cross-platform lineage	✗ Cannot	Lineage beyond Databricks (→ Power BI)	Purview cross-source lineage
Copilot governance	✗ Cannot	Control M365 Copilot data access	Purview DSPM + sensitivity labels
Compliance dashboards	✗ Cannot	Pre-built reports for auditors	Purview built-in compliance dashboards
Data catalog (non-DB)	~ Partial	Only Databricks assets; misses ADLS files + external	Purview cross-source catalog
Business user discovery	~ Partial	Technical UI; no NL search or glossary	Purview Unified Catalog
Privacy / GDPR workflows	~ Partial	Masks PII but cannot find PII estate-wide	Purview auto-detect + rights workflows
Security posture (estate)	~ Partial	Inside Databricks only; no DLP/threat detection	Purview + Defender

7. Minimum Purview Configuration to Close Biggest Gaps

You are already licensed for Purview. If UC-only feels right for now, here is the minimum Purview configuration (2–3 days of setup) that closes your four biggest risks:

Action	What It Closes	Effort	Priority
Configure ADLS scanner	Closes ADLS raw file blind spot; catalogs landing zone	Half a day	Critical
Enable auto-classification on UC assets	Finds PII columns engineers missed; feeds UC masking setup	1 day	Critical
Set up sensitivity label policy	Enables compliance reporting; satisfies GDPR label requirements	1 day	Critical
Create business glossary (top 20 terms)	Closes top business user questions; reduces data team bottleneck	1–2 days	High
Configure cross-platform lineage scan	Connects Databricks assets to Power BI in one lineage graph	Half a day	High
Configure Data Observability view (preview)	Business-friendly quality dashboard for stakeholders	Half a day	Medium

Bottom line: UC gives you operational governance at zero marginal cost. These six Purview configurations (3–5 days total setup) close the compliance, classification, and visibility gaps that UC cannot address. Together they cover the full governance lifecycle without requiring a full Purview rollout.

8. How Both Tools Work Together

The practical integration workflow for your ADLS + Databricks gold layer stack:

Step 1 — Purview classifies → UC enforces

Purview auto-scans your ADLS and UC metadata, identifies PII columns, and applies sensitivity labels. Engineers use those labels as a signal to configure column masking and row filters in UC. Purview finds what needs protecting; UC does the protecting.

Step 2 — UC describes → Purview inherits

Engineers write COMMENT ON TABLE and COMMENT ON COLUMN in UC. Purview picks these up automatically on its next scan and surfaces them to business users in the Unified Catalog. One effort, two audiences.

Step 3 — Purview discovers → UC controls

When a business user finds a data product in Purview and requests access, the approval workflow triggers. The data engineer implements the grant in UC via SQL GRANT. Purview manages the workflow; UC manages the enforcement.

Step 4 — UC audits → Purview reports  [NEW in v4]

UC logs every query in system tables (system.access.audit). For compliance reporting, either query these tables directly in Databricks SQL or forward them to Azure Monitor where Purview’s compliance dashboards present them alongside broader Azure audit data. This gives you granular evidence (UC) with turnkey reporting (Purview).

UC governs what happens to data at runtime. Purview governs what is known about data at design time. Neither replaces the other. Together they cover the full governance lifecycle.

9. Purview Pricing Breakdown  [NEW in v4.1]

Previous versions said ‘pay-per-governed-asset’ without explaining what counts as an asset or estimating cost. Here is the detail.

Pricing Dimension	How It Works	What Counts as an Asset	Estimated Impact (Your Stack)
Scanning	Free since January 2025. No charge to scan ADLS containers, UC metadata, or any connected source.	N/A — scanning itself is free	Zero cost for scanning your ADLS and UC metadata
Governed assets	Charged per unique governed asset per day. An asset is any item Purview catalogs and applies governance to (classification, lineage, glossary link).	Each Delta table = 1 asset. Each ADLS file/folder scanned = 1 asset. Each Power BI dataset = 1 asset. Columns are NOT counted separately.	If you have ~100 gold tables + ~200 ADLS files + ~20 Power BI datasets = ~320 governed assets. At Microsoft’s published rates, this is modest — typically ₹5,000–15,000/month range for a small-to-medium estate.
Data Map population	First 50 governed assets are free. Beyond that, tiered pricing applies. Higher tiers reduce per-asset cost.	Same as above — tables, files, datasets. Not columns or individual records.	Your gold layer (~100 tables) exceeds the free tier. But the cost is low relative to Databricks Premium licensing.
Sensitivity labels	Included in Microsoft 365 E5 or Purview Information Protection P2. No additional per-label cost.	Labels applied to assets — cost is in the license, not per label.	Check your M365 license — E5 includes sensitivity labels. If on E3, additional licensing needed.
Compliance dashboards	Included in Purview governance license. No separate charge for built-in compliance reports.	N/A	No incremental cost once Purview is configured.

Cost reality: Purview’s marginal cost for your stack is likely ₹5,000–15,000/month — modest compared to Databricks Premium licensing. Scanning is free. The real cost is configuration effort (3–5 days), not ongoing charges.

10. Migration Path: UC-Only → Adding Purview Later  [NEW in v4.1]

If you start UC-only and later decide to add Purview, here is exactly what carries over, what needs configuration, and what needs redoing.

What You Did in UC	Carries Over to Purview?	What You Need to Do	Effort
COMMENT ON TABLE/COLUMN	✓ Yes — auto-syncs	Purview scan picks up all descriptions automatically on first scan	Zero — automatic
Tags on tables/columns	✓ Partial — UC tags sync as Purview classifications	Review mapping; some UC tags may not map 1:1 to Purview classification schema	1–2 hours review
Table/schema ownership	✓ Yes — owner metadata syncs	Purview shows UC owners; you can add business contacts on top	Zero — automatic
Access controls (GRANT/REVOKE)	✗ No — UC and Purview access are independent	No action needed — UC continues enforcing. Purview does not replace UC access.	None
Row filters / column masks	✗ No — UC-only, not visible in Purview	No action needed — enforcement stays in UC regardless of Purview	None
Lakehouse Monitoring config	✗ No — UC-only feature	No action needed — quality monitoring continues independently in UC	None
Lineage (auto-captured)	✓ Partial — entity-level lineage syncs; column-level lineage gaps	Purview shows which notebook wrote to which table; full column-level detail stays in UC	Zero — automatic (with known gaps)
Audit logs (system tables)	✗ No — separate systems	Forward UC audit to Azure Monitor if you want unified reporting in Purview dashboards	Half a day for Azure Monitor config
Business glossary	N/A — UC has none	Create glossary from scratch in Purview; link terms to columns already cataloged from UC scan	1–2 days for top 20 terms
Taxonomy / domain hierarchy	N/A — UC has none	Create taxonomy from scratch in Purview; assign UC-scanned assets to domains	1 day
Auto-classification	N/A — UC cannot do	Enable Purview auto-classifiers on UC assets; review results; feed into UC masking decisions	1 day
Sensitivity labels	N/A — UC cannot do	Configure sensitivity label policies; apply to UC-scanned assets	1 day

Migration reality: Starting UC-only costs you nothing when you add Purview later. Your COMMENT ON descriptions, tags, and ownership auto-sync on first Purview scan. Access controls, quality monitoring, and audit logs stay in UC regardless. The only net-new work: glossary, taxonomy, classification, and sensitivity labels — roughly 3–5 days, same as the minimum Purview configuration.

Key takeaway: There is no penalty for starting UC-only. Everything you build in UC carries forward. Purview adds on top — it never requires redoing UC work.

11. Decision Matrix — Quick Reference  [NEW in v4.1]

A simple ‘if X, then do Y’ reference for quick decision-making based on your situation.

If your situation is…	Then use…	Because…
All data in Databricks gold layer, technical team only, low compliance pressure	UC only	UC covers access, lineage, quality, audit completely. No business user discovery needed.
Gold layer + business users asking ‘what does this column mean?’	UC + Purview glossary	Glossary is UC’s biggest gap for business user questions. Minimal Purview config.
Gold layer + GDPR / compliance audit requirements	UC + Purview classification + labels	Auto-classification finds PII you missed. Sensitivity labels satisfy GDPR labeling.
Gold layer + raw ADLS landing zone with sensitive data	UC + Purview ADLS scanner	ADLS raw files are invisible to UC. Purview scanner closes this blind spot.
Gold layer + Power BI dashboards consumed by business	UC + Purview lineage scan	Cross-platform lineage (Databricks → Power BI) only available in Purview.
Gold layer + ML models deployed via MLflow	UC only	Model registry, lineage, feature store, serving audit — all native to UC.
Gold layer + sharing data with external partners	UC only (Delta Sharing)	Delta Sharing is UC-exclusive. No Purview equivalent for live data sharing.
Gold layer + Microsoft 365 Copilot accessing your data	UC + Purview DSPM	DSPM for AI flags which data Copilot can access. UC cannot govern M365 Copilot.
Multiple data sources (ADLS + SQL Server + Power BI + Salesforce)	UC + full Purview	Purview’s 200+ connectors give you a single catalog across all sources.
Small team, low budget, need governance fast	UC only	Included in Premium. Zero marginal cost. Configure in 1–2 days.
Enterprise scale, multiple teams, compliance-heavy	UC + full Purview	Both tools fully configured. UC enforces, Purview discovers and classifies.
Starting today, plan to scale later	UC now → Purview later	No migration penalty. UC work carries forward. Add Purview when needed.

Your situation: Gold layer in Databricks, data in tables, ADLS storage, low business user maturity, both tools licensed. The decision matrix points to: UC now, minimum Purview config (ADLS scanner + auto-classification + sensitivity labels) to close compliance gaps, full Purview when business users start self-serving.

12. Additional Note: Databricks Clean Rooms  [NEW in v4.1]

Related to Data Sharing (Capability 13) but distinct. Databricks Clean Rooms allow two organisations to run joint analysis on combined data without either party seeing the other’s raw data. The compute happens in a secure, audited environment.

Dimension	Databricks Clean Rooms (UC)	Microsoft Purview
Joint analysis without sharing raw data	Yes — both parties contribute data; analysis runs in a secure environment; neither sees the other’s raw records	Not applicable — Purview does not offer a clean room capability
Governed by UC	Yes — UC access controls apply inside the clean room; audit logs capture all activity	N/A
Use case	Two organisations want to match customer records or run joint analytics without exposing PII	N/A
Relevance to your stack	Relevant if you share data with external partners (vendors, advertisers, research collaborators) where raw data cannot be exchanged	N/A — Purview has no equivalent

Note: Clean Rooms complement Delta Sharing. Delta Sharing shares live data with recipients who can see rows. Clean Rooms allow joint computation without either party seeing the other’s raw data. Both are UC-exclusive capabilities with no Purview equivalent.