All authors
Claude Skills by Unknown-333
github.com/Unknown-33337 skills0 installs42 views
- Authoring Airflow DagsWrite production-grade Apache Airflow DAGs using the TaskFlow API — idempotent tasks, correct scheduling and catchup, retries/SLAs, connections/variables, and avoiding top-level code. Use when creating or reviewing Airflow DAGs, scheduling pipelines, wiring task dependencies, configuring retries/backfills, or fixing non-idempotent tasks.Votes: 0GitHub stars: 15
- Building Dagster AssetsBuild Dagster pipelines using software-defined assets — asset dependencies, partitions, resources and IO managers, asset checks, and schedules/sensors. Use when creating Dagster assets or jobs, modeling data as assets, adding partitions or backfills, wiring resources/IO managers, or migrating from task-based orchestration to assets.Votes: 0GitHub stars: 15
- Building Dbt ModelsBuild well-structured dbt models — staging/intermediate/marts layers, ref() and source(), materializations, and incremental models with the right strategy. Use when creating or refactoring dbt models, choosing table vs view vs incremental, structuring a dbt project, or writing incremental logic.Votes: 0GitHub stars: 15
- Building Feature PipelinesBuild ML feature pipelines and feature stores — point-in-time-correct joins to avoid label leakage, offline/online parity, feature freshness and backfills, and materialization with tools like Feast. Use when engineering features for ML, preventing train/serve skew or data leakage, building a feature store, or backfilling historical features for training.Votes: 0GitHub stars: 15
- Building Iceberg TablesDesign and operate Apache Iceberg tables — partitioning and hidden partitioning, partition/schema evolution, snapshots and time travel, compaction and small-file cleanup, and MERGE/upsert for lakehouse tables on Spark, Flink, Trino, or Snowflake. Use when creating or maintaining Iceberg tables, choosing partitioning, evolving schema/partitions, or fixing small-file and metadata bloat.Votes: 0GitHub stars: 15
- Building Ingestion PipelinesBuild batch and incremental data ingestion (extract-load) pipelines — full vs incremental extraction, change data capture (CDC), watermarks and high-water marks, API pagination and rate limits, and choosing managed EL tools (Fivetran, Airbyte) vs custom code. Use when ingesting data from databases, APIs, files, or SaaS into a warehouse/lake, or designing incremental extraction and CDC.Votes: 0GitHub stars: 15
- Building Kafka ConsumersBuild reliable Apache Kafka consumers and producers — consumer groups and partition assignment, offset commit strategy, at-least-once vs exactly-once, idempotent/transactional producers, rebalancing, and dead-letter handling. Use when writing Kafka consumers/producers, configuring offset commits or consumer groups, tuning throughput, or handling rebalances and poison messages.Votes: 0GitHub stars: 15
- Debugging Airflow PipelinesDiagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.Votes: 0GitHub stars: 15
- Debugging Data PipelinesSystematically root-cause data pipeline failures and data incidents — job errors, wrong or missing data, duplicates, and freshness misses — by tracing lineage upstream, isolating the failing stage, reconciling against source, and planning a safe fix and backfill. Use when a pipeline fails, numbers look wrong, data is missing or duplicated, a dashboard is stale, or a stakeholder reports a data discrepancy.Votes: 0GitHub stars: 15
- Debugging Dbt RunsDiagnose and fix failing dbt runs — Jinja/compilation errors, ref/dependency and DAG issues, incremental models producing wrong or duplicate rows, full-refresh needs, and state/deferral problems. Use when dbt run or dbt build fails, a model compiles wrong, an incremental model is stale or duplicated, or a CI dbt job errors.Votes: 0GitHub stars: 15
- Designing Backfills And ReplaysPlan and run safe data backfills and replays — idempotent reprocessing of historical windows, partition-by-partition execution, isolating backfill compute from production, verifying results, and avoiding double-counting or changed history. Use when backfilling a new or fixed model, reprocessing after a bug, replaying events, or loading history for a new pipeline without corrupting existing data.Votes: 0GitHub stars: 15
- Designing Data ContractsDefine and enforce data contracts between producers and consumers — explicit schema, semantics, ownership, SLAs, and versioning — to prevent silent upstream changes from breaking downstream pipelines. Use when a producer schema change could break consumers, defining an interface between teams/services and the warehouse, or adding schema enforcement at ingestion.Votes: 0GitHub stars: 15
- Designing Medallion ArchitectureStructure a lakehouse with the medallion architecture — bronze (raw), silver (cleaned/conformed), and gold (business/aggregated) layers — with clear responsibilities, idempotent layer transitions, and where to put quality checks and modeling. Use when organizing a data lakehouse, defining bronze/silver/gold layers, deciding what logic belongs in each layer, or refactoring a flat pipeline into layers.Votes: 0GitHub stars: 15
- Documenting Dbt ModelsDocument a dbt project — model and column descriptions in schema.yml, source definitions, exposures, doc blocks, and generated docs/lineage. Use when adding descriptions to dbt models, documenting sources or dashboards as exposures, setting up dbt docs, or improving data catalog coverage.Votes: 0GitHub stars: 15
- Engineering Databricks PipelinesBuild reliable Databricks pipelines on the lakehouse — Delta Lake tables, MERGE and time travel, Delta Live Tables / Lakeflow declarative pipelines, Unity Catalog governance, Photon, Auto Loader ingestion, and cluster/job sizing. Use when building Databricks jobs or DLT pipelines, working with Delta tables, ingesting with Auto Loader, or organizing Unity Catalog.Votes: 0GitHub stars: 15
- Generating Synthetic Test DataGenerate realistic synthetic data for testing data pipelines — deterministic seeded fixtures, referential integrity across tables, edge cases (nulls, duplicates, late/out-of-order events), volume for load tests, and privacy-safe stand-ins for production. Use when creating test data for pipeline/dbt tests, seeding dev environments, load testing, or replacing PII with safe synthetic data.Votes: 0GitHub stars: 15
- Handling Schema EvolutionEvolve data schemas safely over time — backward/forward compatibility, additive vs breaking changes, column adds/renames/type changes, and evolution in Avro, Parquet, Iceberg, Delta, and warehouse tables. Use when changing a table or event schema, adding or renaming columns, changing types, or preventing a schema change from breaking readers or pipelines.Votes: 0GitHub stars: 15
- Implementing Cdc With DebeziumCapture database changes with Debezium change data capture — connector setup for Postgres/MySQL/SQL Server, snapshot vs streaming phases, handling inserts/updates/deletes and tombstones, schema changes, and applying the change stream idempotently to a warehouse/lake. Use when setting up CDC, replicating an OLTP database, capturing deletes, or consuming a Debezium change stream.Votes: 0GitHub stars: 15
- Implementing Data CicdSet up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.Votes: 0GitHub stars: 15
- Implementing Data Quality ChecksAdd data quality checks to pipelines — freshness, volume/row-count anomalies, schema drift, null/uniqueness/referential integrity, and value distributions — using dbt tests, Great Expectations, or Soda, and deciding warn vs block. Use when adding data quality validation, catching bad data before it reaches consumers, setting up freshness/volume checks, or defining expectations.Votes: 0GitHub stars: 15
- Implementing Pipeline ObservabilityInstrument data pipelines with observability — freshness and volume SLAs, run success/latency metrics, data quality monitors, anomaly detection, lineage, and actionable alerting that avoids fatigue. Use when pipelines fail silently, incidents are found by stakeholders instead of alerts, setting up monitoring/SLAs for data, or reducing noisy alerts.Votes: 0GitHub stars: 15
- Managing Data Lineage OpenlineageCapture and use data lineage with OpenLineage and Marquez — emitting run/job/dataset events from Airflow, dbt, and Spark, column-level lineage, and using lineage for impact analysis, debugging, and backfill scoping. Use when setting up data lineage, integrating OpenLineage, tracing what a change breaks downstream, or scoping the blast radius of an incident.Votes: 0GitHub stars: 15
- Masking Pii DataProtect personally identifiable information in data pipelines — classifying PII, choosing masking vs tokenization vs hashing vs encryption, dynamic data masking and column-level access control, and handling deletion/right-to-be-forgotten. Use when handling sensitive data, masking or anonymizing PII, meeting GDPR/CCPA/HIPAA requirements, or restricting column access in a warehouse.Votes: 0GitHub stars: 15
- Migrating Legacy EtlPlan and execute migrations of legacy ETL and data warehouses — stored procedures, SSIS/Informatica, or on-prem warehouses to modern stacks (dbt, Spark, cloud warehouses) — using strangler-fig phasing, parallel runs, and row/aggregate reconciliation. Use when migrating legacy pipelines or warehouses, rewriting stored procedures into dbt/Spark, or validating a migration against the source system.Votes: 0GitHub stars: 15
- Modeling Dimensional DataDesign analytics data models using dimensional modeling — star and snowflake schemas, fact and dimension tables, grain declaration, surrogate keys, and slowly changing dimensions (SCD Type 1/2/3). Use when designing a warehouse schema, building marts, choosing a table grain, tracking history, or deciding fact vs dimension.Votes: 0GitHub stars: 15
- Optimizing Bigquery QueriesReduce Google BigQuery cost and runtime — partitioning and clustering, minimizing bytes processed, avoiding SELECT * and full scans, slot usage and reservations, approximate functions, and materialized views. Use when BigQuery queries are expensive or slow, bytes billed are high, a query scans full tables, or you need to size slots.Votes: 0GitHub stars: 15
- Optimizing Parquet StorageOptimize columnar Parquet storage for analytics — file and row-group sizing, compression codecs (Snappy/ZSTD), partitioning and file layout, column pruning and predicate pushdown, dictionary encoding, and fixing the small-files problem. Use when Parquet reads are slow or costly, files are too small/large, choosing compression or partitioning, or improving scan pruning on a data lake.Votes: 0GitHub stars: 15
- Optimizing Pyspark JobsOptimize slow or failing PySpark and Spark SQL jobs — partitioning and repartitioning, data skew, shuffles, broadcast joins, caching, Adaptive Query Execution, and avoiding driver collects and Python UDFs. Use when a Spark job is slow, spills, OOMs, has skewed tasks, runs a huge shuffle, or a stage hangs on a few straggler tasks.Votes: 0GitHub stars: 15
- Optimizing Snowflake WorkloadsReduce Snowflake cost and latency — right-size and auto-suspend warehouses, use multi-cluster for concurrency, apply clustering keys, read the Query Profile, exploit result/warehouse caching, and control credit spend. Use when Snowflake queries are slow or expensive, warehouses spill or queue, credits are high, or you need to size a warehouse.Votes: 0GitHub stars: 15
- Optimizing Sql QueriesOptimize slow analytical SQL by reading query/EXPLAIN plans, cutting scanned data, fixing join strategy, and using partitioning, clustering, and indexes across Postgres, Snowflake, BigQuery, Databricks/Spark SQL, and Redshift. Use when a query is slow, times out, costs too much, scans too many rows/bytes, or spills to disk.Votes: 0GitHub stars: 15
- Orchestrating Prefect FlowsBuild Prefect workflows — flows and tasks, retries and caching, parameters, concurrency limits, deployments and schedules, blocks for connections/secrets, and idempotent task design. Use when writing or debugging Prefect flows, scheduling runs, configuring retries/caching, or migrating scripts to Prefect orchestration.Votes: 0GitHub stars: 15
- Processing Streaming DataBuild reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.Votes: 0GitHub stars: 15
- Reviewing Data Pipeline CodeReview data engineering pull requests with a data-specific checklist — idempotency, correct grain, incremental logic, cost impact, data quality tests, PII handling, and backward compatibility — that generic code review misses. Use when reviewing a dbt/SQL/Spark/Airflow PR, a data pipeline change, or a new model, and you want to catch data-correctness and cost issues before merge.Votes: 0GitHub stars: 15
- Terraform For Data InfraProvision data infrastructure with Terraform — warehouses, buckets, IAM/roles, orchestration, and streaming resources — using modules, remote state, workspaces/environments, and safe plan/apply workflows. Use when writing Terraform for Snowflake/BigQuery/Redshift, S3/GCS, IAM, Airflow/MWAA, or Kafka, or structuring reusable data-infra modules across environments.Votes: 0GitHub stars: 15
- Testing Dbt ProjectsAdd data quality tests to a dbt project — generic tests (unique, not_null, accepted_values, relationships), singular tests, unit tests, dbt-utils and dbt-expectations packages, and source freshness. Use when adding tests to dbt models, catching data quality regressions, validating assumptions, or setting up source freshness checks.Votes: 0GitHub stars: 15
- Writing Idempotent TransformationsWrite data transformations and loads that produce the same result no matter how many times they run — using MERGE/upsert, deterministic partition overwrites, deduplication, and stable keys. Use when a retry could duplicate data, a job is not safe to re-run, a pipeline needs exactly-once effects, or loads must be backfill-safe.Votes: 0GitHub stars: 15
- Skill TemplateTemplate for authoring a new data-engineering skill. Replace this text with a third-person summary of what the skill does and when to use it, including concrete trigger keywords (tools, file types, tasks). Copy this folder to skills/<your-skill-name>/ and rename it.Votes: 0GitHub stars: 15