Informatica

---
name: informatica
description: "Informatica IDMC specialist for enterprise cloud data integration, quality, governance, and application integration. Deep expertise in CDI mappings, pushdown optimization, Secure Agents, taskflows, CLAIRE AI, CDC, and migration from PowerCenter. WHEN: \"Informatica\", \"IDMC\", \"Informatica Cloud\", \"IICS\", \"CDI\", \"Informatica mapping\", \"Informatica mapplet\", \"Informatica pushdown\", \"pushdown optimization\", \"Secure Agent\", \"Informatica taskflow\", \"CLAIRE\", \"CLAIRE GPT\", \"Informatica connector\", \"Informatica CDC\", \"mass ingestion\", \"Informatica IPU\", \"PowerCenter migration\", \"Informatica serverless\", \"CDI-Elastic\", \"Informatica data quality\", \"CDQ\", \"CDGC\", \"Informatica lookup\", \"dynamic mapping task\"."
license: MIT
---

# Informatica IDMC

This skill covers Informatica Intelligent Data Management Cloud (IDMC), a comprehensive AI-powered cloud-native platform for enterprise data management. IDMC is a managed platform with continuous releases (seasonal naming: Spring, Summer, Fall). It provides deep knowledge of:

- Cloud Data Integration (CDI) mappings, transformations, and execution modes
- Pushdown Optimization (ELT) for source-side, target-side, and full pushdown
- Secure Agent architecture, groups, and high availability
- Serverless runtime (Standard and Advanced Serverless)
- Taskflow orchestration (standard and linear)
- CLAIRE AI engine, Copilot, GPT, and CLAIRE Agents
- Change Data Capture (CDC) and Mass Ingestion
- Data Quality (CDQ), Data Governance and Catalog (CDGC), and MDM
- Connector ecosystem (300+ native, 10,000+ metadata-aware)
- CI/CD via Git integration and REST APIs
- Migration from PowerCenter to IDMC

## How to Approach Tasks

When you receive a request:

1. **Classify** the request:
   - **Architecture / mapping design** -- Load `references/architecture.md` for CDI, Secure Agent, serverless runtime, taskflow orchestration, CLAIRE AI, and connector ecosystem
   - **Performance / best practices** -- Load `references/best-practices.md` for mapping optimization, pushdown tuning, error handling, CI/CD, security, and cost management
   - **Troubleshooting / diagnostics** -- Load `references/diagnostics.md` for agent connectivity, mapping errors, session failures, performance bottlenecks, and monitoring
   - **PowerCenter migration** -- Consult architecture reference for Migration Factory guidance, then apply IDMC-specific patterns
   - **Cross-tool comparison** -- See the `integration` skill for Informatica vs ADF, NiFi, Fivetran, etc.

2. **Gather context** -- Determine:
   - What is the integration doing? (batch ETL, ELT pushdown, CDC, real-time, API integration)
   - What runtime? (Secure Agent, CDI-Elastic, Advanced Serverless)
   - Is this development, test, or production?
   - Is there existing PowerCenter infrastructure being migrated?

3. **Analyze** -- Apply IDMC-specific reasoning. Consider pushdown optimization eligibility, Secure Agent group topology, IPU consumption, CLAIRE recommendations, and taskflow orchestration patterns.

4. **Recommend** -- Provide actionable guidance with specific transformation names, pushdown configuration, taskflow step types, REST API endpoints, and CLAIRE capabilities where appropriate.

5. **Verify** -- Suggest validation steps (Activity Monitor review, session log thread statistics, pushdown SQL inspection, Secure Agent health check).

## Core Architecture

### CDI Mapping-Transformation-Connection Model

```
┌──────────────────────────────────────────────────┐
│  Mapping                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │  Source  │──│  Filter  │──│  Expression  │  │
│  │          │  │          │  │              │  │
│  └──────────┘  └──────────┘  └──────┬───────┘  │
│                                     │           │
│                              ┌──────▼──────┐    │
│                              │   Lookup    │    │
│                              └──────┬──────┘    │
│                              ┌──────▼──────┐    │
│                              │   Target    │    │
│                              └──────┬──────┘    │
└─────────────────────────────────────┼───────────┘
                                      │
                            ┌─────────▼─────────┐
                            │  Runtime Engine    │
                            │ (Agent/Serverless) │
                            └─────────┬─────────┘
                                      │
                            ┌─────────▼─────────┐
                            │   Data Source      │
                            └───────────────────┘
```

**Mappings** are the primary design artifact. Each mapping defines sources, targets, transformations, and parameterization. Mappings are executed as mapping tasks on a selected runtime environment.

**Transformations** (30+ built-in): Filter, Router, Joiner, Union, Sorter, Expression, Aggregator, Normalizer, Lookup (connected/unconnected), Sequence Generator, Hierarchy Processor, Java Transformation, Python Transformation, SQL Transformation, Data Masking, Rank.

**Mapplets** encapsulate reusable transformation logic with Input/Output transformations. Mapplets can be nested (non-cyclically) and parameterized for maximum reuse. Version independently from consuming mappings.

**Connections** define authentication and endpoints for data stores and applications. Support parameterization for environment-specific resolution.

**Dynamic Mapping Tasks** reduce asset proliferation by defining multiple jobs with different source/target configurations within a single parameterized mapping.

### Runtime Engines

| Runtime | Infrastructure | Compute | Best For |
|---|---|---|---|
| **Secure Agent** | Customer-managed (Windows/Linux) | Single server | On-premises sources, hybrid connectivity, full control |
| **CDI-Elastic** | Customer-managed Spark clusters | Auto-scaling Spark | Large-scale distributed processing |
| **Advanced Serverless** | Informatica-managed | Fully serverless, auto-scaling to zero | Cloud-to-cloud integrations, no infrastructure management |

**Secure Agents** are lightweight programs installed on-premises or in customer-managed cloud VMs. They bridge IDMC cloud services with local data sources via encrypted outbound HTTPS communication. Agents run multiple microservices: Data Integration Server, Metadata Agent Service, Process Server, Mass Ingestion Service, API Gateway Agent.

**Secure Agent Groups** contain multiple agents for workload distribution, department isolation, and environment separation. HA is achieved via multi-agent groups with failover.

**Advanced Serverless** provisions compute in an INFA DMZ adjacent to the customer VPC, connected via tenant-controlled Elastic Network Interfaces (ENIs). CLAIRE ML engine auto-tunes job performance. AWS primary; Azure support added recently.

### Pushdown Optimization (ELT)

Pushdown converts transformation logic to SQL and pushes execution to the database engine:

| Mode | Behavior | Best For |
|---|---|---|
| **Source-Side** | Pushes logic to source database, reducing data read | Filters and aggregations that reduce volume |
| **Target-Side** | Pushes logic to target database via INSERT/DELETE/UPDATE | Complex transforms expressible in target-native SQL |
| **Full Pushdown** | All logic pushed to target; falls back to source-side if partial | Source and target on same platform |

Supported transformations: Filter, Aggregator, Expression, Joiner, Sorter, Union, Lookup (with restrictions), Sequence Generator (temporary sequence).

Limitations: Variable ports in Expression transformations not supported. Database-specific functions may lack SQL equivalents. Lookup with multiple match policies must use "Report Error". SQL override queries require "Create Temporary View" enabled.

### Taskflow Orchestration

**Standard Taskflow**: Full orchestration with parallel execution, decision logic, scheduling, and recovery from failure point.

**Linear Taskflow**: Simplified sequential execution. If a task fails, the entire workflow must restart (no resume-from-failure).

**13 Step Types**: Assignment, Data Task, Notification Task, Command Task, File Watch Task, Ingestion Task, Subtaskflow, Decision, Parallel Paths, Jump (looping), Wait, Throw (fault handling), End.

**Execution methods**: Designer UI, REST/SOAP APIs, RunAJob utility, file listener events, scheduled execution.

**Publishing**: REST/SOAP Binding (APIs with access controls), Event Binding (file arrival triggers), Schedule Binding (automated periodic execution).

### CLAIRE AI Engine

CLAIRE (Cloud-scale AI for Real-time Execution) operates across three tiers:

1. **CLAIRE AI Engine**: Core ML layer -- metadata intelligence, auto-mapping, data quality suggestions, anomaly detection, auto-tuning for serverless
2. **CLAIRE Copilot**: Assistive AI in developer workflows -- context-aware suggestions, best practice recommendations, performance optimization hints
3. **CLAIRE GPT**: Conversational AI -- natural language data management, integrated with Azure OpenAI and Anthropic Claude, agentic data management with planning and reasoning

**CLAIRE Agents** (Fall 2025+): Data Exploration Agents, Enterprise Discovery Agents, ELT Agents for business-user pipeline creation, AI Agent Engineering (private preview) for no-code agent building.

### Connector Ecosystem

- **300+ native connectors** with built-in governance
- **10,000+ metadata-aware connectors** across AWS, Azure, GCP
- **Categories**: Relational databases (Oracle, SQL Server, PostgreSQL, MySQL, DB2), cloud warehouses (Snowflake, Redshift, BigQuery, Synapse), cloud storage (S3, Azure Blob, GCS), SaaS (Salesforce, SAP, Workday, ServiceNow), messaging (Kafka, Kinesis, Event Hubs), files (CSV, JSON, Parquet, Avro, ORC), APIs (REST, SOAP, OData)
- **GenAI connectors** (Summer 2025+): NVIDIA NIM, Databricks Mosaic AI, Snowflake Cortex AI
- **MCP support**: Connect AI agents and LLMs to IDMC assets as tools

### CDC and Mass Ingestion

- **Log-based CDC**: Minimal source system impact with exactly-once database replication guarantees
- **Initial load + incremental CDC**: Support for combined patterns
- **Schema drift handling**: Automatic detection and adaptation to source schema changes
- **Mass Ingestion**: Petabyte-scale bulk data movement with wizard-driven job creation
- **SuperPipe**: Optimized streaming into Snowflake via Snowpipe Streaming
- **Supported CDC sources**: Oracle, SQL Server, MySQL, PostgreSQL, DB2

### Additional Platform Services

| Service | Capability |
|---|---|
| **Cloud Data Quality (CDQ)** | Profiling, cleansing, standardization, fuzzy matching, quality scorecards, DQ Rules as API |
| **Data Governance and Catalog (CDGC)** | AI-powered catalog, business glossary, end-to-end lineage, data marketplace, compliance |
| **Master Data Management (MDM)** | Golden record management, match/merge, hierarchy management, stewardship workflows |
| **Cloud Application Integration (CAI)** | Process automation, service connectors, event-driven triggers, B2B/EDI integration |
| **API Manager** | Full API lifecycle: create, publish, manage, monitor, deprecate |

### Pricing Model

Informatica Processing Units (IPU) -- unified consumption-based metric across all IDMC services. 100% usage-based pricing. Provides flexibility to reallocate compute across services as needs evolve.

### Monitoring

- **Activity Monitor**: Real-time job tracking with filtering by date, type, status, runtime, user
- **Session logs**: Thread statistics (reader/transformation/writer busy percentages) for bottleneck identification
- **IDMC Log Analyzer**: IPU consumption tracking, bottleneck identification, audit trail analysis
- **External integration**: Dynatrace extension, Splunk/Datadog log forwarding, REST API-based custom monitoring
- **Secure Agent Health Check**: Informatica-provided accelerator for agent configuration and connectivity review

## Anti-Patterns

1. **Ignoring pushdown optimization** -- Processing data in-memory on the Secure Agent when the database engine could handle it. Enable pushdown for Filter, Aggregator, and Expression transformations to reduce data movement and IPU consumption.
2. **Unnecessary data type conversions** -- Each CAST operation adds overhead, especially in pushdown where extra CAST statements are generated in SQL. Maintain consistent types from source through target.
3. **Using Java Transformation for simple logic** -- Java transformations have higher startup costs. Use Expression transformation for simple calculations and string operations.
4. **Linear taskflows for critical workflows** -- Linear taskflows cannot resume from failure point. Use standard taskflows for production workflows that need recovery capability.
5. **Hardcoding environment values in mappings** -- Use Parameter Contexts and connection parameterization. Dynamic Mapping Tasks for multi-source/target patterns.
6. **Oversized lookup caches** -- Large lookup tables cached entirely in memory exhaust agent heap. Use database joins or pushdown lookups for large reference data.
7. **Ignoring thread statistics** -- Session logs contain thread statistics (reader, transformation, writer busy percentages) that pinpoint the exact bottleneck. The thread with the highest busy percentage is the bottleneck.
8. **Single Secure Agent with no HA** -- Production environments need multi-agent groups for failover. A single agent is a single point of failure.

## Reference Files

- `references/architecture.md` -- CDI mapping model, Secure Agent internals, serverless runtime, CLAIRE AI tiers and agents, taskflow orchestration, connector ecosystem, CDC and mass ingestion, deployment model
- `references/best-practices.md` -- Mapping design (transformation selection, pushdown configuration, reusable mapplets, parameterization), performance tuning (thread statistics, partitioning, source/target optimization), error handling, CI/CD workflow, security, cost optimization
- `references/diagnostics.md` -- Secure Agent connectivity, mapping errors, session failures, taskflow failures, performance diagnostics (thread statistics, bottleneck identification, pushdown analysis), monitoring (Activity Monitor, Log Analyzer, alerting), diagnostic workflow

## Cross-References

- `integration` skill -- Informatica vs ADF, NiFi, Fivetran comparisons
- `overview` skill -- Cross-tool comparisons and paradigm routing