Privacy Data Sharing

---
name: privacy-data-sharing
description: >-
  Build privacy-preserving data sharing platforms using synthetic data generation
  with the SDV library, data clean rooms, secure enclaves, and utility
  measurement. Covers end-to-end architecture for sharing analytical datasets
  while preserving individual privacy guarantees.
license: Apache-2.0
metadata:
  author: mukul975
  version: "1.0"
  domain: privacy
  subdomain: privacy-engineering
  tags: "data-sharing, synthetic-data, data-clean-rooms, secure-enclaves, sdv-library"
---

# Privacy-Preserving Data Sharing Platform

## Overview

Privacy-preserving data sharing enables organizations to derive analytical value from combined datasets without exposing raw personal data. This skill covers four primary approaches: synthetic data generation, data clean rooms, secure enclaves, and federated analytics, along with utility measurement frameworks to ensure shared data remains useful.

## Approach Selection Framework

| Approach | Privacy Guarantee | Data Utility | Computational Cost | Trust Model |
|----------|-------------------|-------------|-------------------|-------------|
| Synthetic Data | Statistical (configurable) | High for distributions, lower for edge cases | Medium (training) | No trust required |
| Data Clean Rooms | Contractual + technical | High (real data, restricted queries) | Low-Medium | Trusted operator |
| Secure Enclaves (TEE) | Hardware-backed isolation | Very high (real data) | Medium | Trust hardware vendor |
| Federated Analytics | Cryptographic/DP | Medium-High | High (communication) | Minimal trust |
| Homomorphic Encryption | Cryptographic | High | Very High | No trust required |
| Secure Multi-Party Computation | Cryptographic | High | High | Honest majority |

## Synthetic Data Generation with SDV

### Architecture

```
Source Data --> Statistical Profiling --> Model Training --> Synthetic Generation
                    |                        |                      |
                    v                        v                      v
            Metadata Analysis        Model Selection         Quality Assessment
            - Column types           - GaussianCopula        - Statistical tests
            - Distributions          - CTGAN                 - Privacy metrics
            - Correlations           - CopulaGAN             - Utility metrics
            - Constraints            - TVAE                  - Visual comparison
```

### SDV Implementation

```python
"""
Synthetic data generation using the Synthetic Data Vault (SDV) library.
Generates privacy-preserving synthetic datasets that maintain statistical
properties of the original data.
"""

import pandas as pd
import numpy as np
from sdv.metadata import SingleTableMetadata
from sdv.single_table import GaussianCopulaSynthesizer, CTGANSynthesizer, TVAESynthesizer
from sdv.evaluation.single_table import run_diagnostic, evaluate_quality
from sdmetrics.reports.single_table import QualityReport


def create_metadata(df: pd.DataFrame) -> SingleTableMetadata:
    """Auto-detect and create metadata for a DataFrame."""
    metadata = SingleTableMetadata()
    metadata.detect_from_dataframe(df)
    return metadata


def train_gaussian_copula(
    df: pd.DataFrame,
    metadata: SingleTableMetadata,
    enforce_min_max: bool = True
) -> GaussianCopulaSynthesizer:
    """
    Train a Gaussian Copula model for synthetic data generation.
    Best for: Datasets with mostly numerical data and linear correlations.
    """
    synthesizer = GaussianCopulaSynthesizer(
        metadata,
        enforce_min_max_values=enforce_min_max,
        numerical_distributions={
            "norm": "beta",  # Fit beta distributions for bounded numerical data
        }
    )
    synthesizer.fit(df)
    return synthesizer


def train_ctgan(
    df: pd.DataFrame,
    metadata: SingleTableMetadata,
    epochs: int = 300,
    batch_size: int = 500
) -> CTGANSynthesizer:
    """
    Train a CTGAN model for synthetic data generation.
    Best for: Complex distributions, mixed data types, mode-specific patterns.
    """
    synthesizer = CTGANSynthesizer(
        metadata,
        epochs=epochs,
        batch_size=batch_size,
        verbose=True
    )
    synthesizer.fit(df)
    return synthesizer


def train_tvae(
    df: pd.DataFrame,
    metadata: SingleTableMetadata,
    epochs: int = 300
) -> TVAESynthesizer:
    """
    Train a TVAE model for synthetic data generation.
    Best for: Datasets where CTGAN struggles, faster training than CTGAN.
    """
    synthesizer = TVAESynthesizer(
        metadata,
        epochs=epochs
    )
    synthesizer.fit(df)
    return synthesizer


def generate_synthetic_data(
    synthesizer,
    num_rows: int
) -> pd.DataFrame:
    """Generate synthetic data from a trained synthesizer."""
    return synthesizer.sample(num_rows=num_rows)


def evaluate_synthetic_quality(
    real_data: pd.DataFrame,
    synthetic_data: pd.DataFrame,
    metadata: SingleTableMetadata
) -> dict:
    """
    Evaluate the quality of synthetic data against real data.

    Returns diagnostic and quality scores.
    """
    # Run diagnostic checks
    diagnostic = run_diagnostic(
        real_data=real_data,
        synthetic_data=synthetic_data,
        metadata=metadata
    )

    # Run quality evaluation
    quality = evaluate_quality(
        real_data=real_data,
        synthetic_data=synthetic_data,
        metadata=metadata
    )

    return {
        "diagnostic_score": diagnostic.get_score(),
        "quality_score": quality.get_score(),
    }


def measure_privacy_risk(
    real_data: pd.DataFrame,
    synthetic_data: pd.DataFrame,
    metadata: SingleTableMetadata,
    key_fields: list[str]
) -> dict:
    """
    Measure re-identification risk in synthetic data.

    Checks for exact matches and nearest-neighbor distances
    between real and synthetic records.
    """
    # Check for exact record matches
    merged = real_data.merge(synthetic_data, how="inner")
    exact_match_rate = len(merged) / len(real_data)

    # Check key field matches
    if key_fields:
        key_merged = real_data[key_fields].merge(
            synthetic_data[key_fields], how="inner"
        )
        key_match_rate = len(key_merged) / len(real_data)
    else:
        key_match_rate = 0.0

    return {
        "exact_match_rate": exact_match_rate,
        "key_match_rate": key_match_rate,
        "privacy_safe": exact_match_rate < 0.01 and key_match_rate < 0.05,
    }
```

### Model Selection Guide

| Factor | GaussianCopula | CTGAN | TVAE |
|--------|---------------|-------|------|
| Training speed | Fast (minutes) | Slow (hours) | Medium (30-60 min) |
| Small datasets (<1K rows) | Good | Poor | Fair |
| Large datasets (>100K rows) | Good | Good | Good |
| Numerical data | Excellent | Good | Good |
| Categorical data (high cardinality) | Fair | Good | Good |
| Complex correlations | Fair | Good | Good |
| Constraint handling | Good | Fair | Fair |
| Reproducibility | Excellent | Fair (seed-dependent) | Fair |

## Data Clean Room Architecture

### Components

```
Organization A                    Clean Room                    Organization B
+-------------+    encrypted    +------------------+    encrypted    +-------------+
| Source Data  |  ----------->  | Ingestion Layer  |  <-----------  | Source Data  |
+-------------+                 +------------------+                 +-------------+
                                        |
                                        v
                                +------------------+
                                | Data Preparation |
                                | - Schema mapping |
                                | - Normalization  |
                                | - Deduplication  |
                                +------------------+
                                        |
                                        v
                                +------------------+
                                | Approved Queries |
                                | - Pre-approved   |
                                |   query templates|
                                | - Aggregate only |
                                | - Min group size |
                                +------------------+
                                        |
                                        v
                                +------------------+
                                | Output Validation|
                                | - k-anonymity    |
                                | - DP noise       |
                                | - Disclosure risk|
                                +------------------+
                                        |
                            +-----------+-----------+
                            |                       |
                            v                       v
                    Results for Org A        Results for Org B
```

### Clean Room Policy Engine

```python
"""
Policy engine for data clean room query validation.
Enforces privacy rules on all queries before execution.
"""

from dataclasses import dataclass


@dataclass
class CleanRoomPolicy:
    min_group_size: int = 50
    allowed_operations: list[str] = None
    blocked_columns: list[str] = None
    max_output_rows: int = 1000
    require_aggregation: bool = True
    dp_epsilon: float = 1.0

    def __post_init__(self):
        if self.allowed_operations is None:
            self.allowed_operations = ["COUNT", "SUM", "AVG", "MEDIAN", "PERCENTILE"]
        if self.blocked_columns is None:
            self.blocked_columns = ["ssn", "email", "phone", "full_name", "address"]


class QueryValidator:
    """Validate clean room queries against privacy policies."""

    def __init__(self, policy: CleanRoomPolicy):
        self.policy = policy

    def validate(self, query_ast: dict) -> tuple[bool, list[str]]:
        """
        Validate a parsed query against the policy.

        Returns (is_valid, list_of_violations).
        """
        violations = []

        # Check for blocked columns
        referenced_columns = query_ast.get("columns", [])
        for col in referenced_columns:
            if col.lower() in self.policy.blocked_columns:
                violations.append(f"Column '{col}' is blocked by policy")

        # Check aggregation requirement
        if self.policy.require_aggregation:
            if not query_ast.get("has_aggregation", False):
                violations.append("Query must include aggregation (no raw record output)")

        # Check operations
        operations = query_ast.get("operations", [])
        for op in operations:
            if op.upper() not in self.policy.allowed_operations:
                violations.append(f"Operation '{op}' is not in allowed operations list")

        # Check output size
        if query_ast.get("limit", float("inf")) > self.policy.max_output_rows:
            violations.append(
                f"Output exceeds max rows ({self.policy.max_output_rows})"
            )

        return (len(violations) == 0, violations)
```

## Secure Enclave Integration

### Intel SGX / Azure Confidential Computing

```
Data Owner A           Confidential Computing          Data Owner B
                       +----------------------+
Data (encrypted) ----> | Enclave Environment  | <---- Data (encrypted)
                       | - Decryption in TEE  |
                       | - Join/Analysis      |
                       | - Re-encrypt results |
                       +----------------------+
                               |
                       Encrypted Results
                       (only to authorized parties)
```

### Key Properties
- **Confidentiality**: Data is encrypted outside the enclave; only decrypted within the TEE
- **Integrity**: Enclave code is measured and attested; tampering is detectable
- **Attestation**: Remote parties can verify the enclave is running approved code
- **Isolation**: Even the cloud provider cannot access data inside the enclave

## Utility Measurement Framework

### Statistical Utility Metrics

| Metric | Description | Target |
|--------|-------------|--------|
| Column Shapes | Distribution similarity per column (KS test) | > 0.85 |
| Column Pair Trends | Correlation preservation between column pairs | > 0.80 |
| Boundary Adherence | Values within real data min/max ranges | > 0.95 |
| Category Coverage | All categories in real data appear in synthetic | > 0.90 |
| Range Coverage | Numeric ranges adequately represented | > 0.85 |

### Privacy Metrics

| Metric | Description | Target |
|--------|-------------|--------|
| Exact Match Rate | % of synthetic records identical to real records | < 1% |
| Nearest Neighbor Distance | Minimum distance from synthetic to nearest real record | > threshold |
| Membership Inference AUC | Ability of attack model to determine membership | < 0.55 |
| Attribute Inference Accuracy | Ability to infer sensitive attributes | < random + 5% |
| k-Anonymity of output | Minimum equivalence class size | k >= 5 |

## References

- Patki, N., Wedge, R., and Veeramachaneni, K. "The Synthetic Data Vault." IEEE DSAA, 2016.
- SDV Documentation: docs.sdv.dev
- Xu, L. et al. "Modeling Tabular Data Using Conditional GAN." NeurIPS, 2019.
- Google BigQuery Clean Rooms Documentation
- AWS Clean Rooms Service Documentation
- Microsoft Azure Confidential Computing Documentation
- Stadler, T. et al. "Synthetic Data — Anonymisation Groundhog Day." USENIX Security, 2022.