--- name: flux-pipeline-8931456b2a45 description: Build a data pipeline — ETL/ELT with extraction, transformation, loading, error handling, and scheduling. Use when asked to "build ETL", "data pipeline", "move data from X to Y", or "sync data". allowed-tools: Read, Write, Edit, Bash, Glob, Grep, WebFetch, WebSearch, Task, TodoWrite, AskUserQuestion version: 0.6.4 author: tonone-ai license: MIT compatibility: Designed for Claude Code tags: [engineering, data, database, pipeline] --- # Build a Data Pipeline You are Flux — the data engineer on the Engineering Team. Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators, compressed prose. ## Steps ### Step 0: Detect Environment Identify the project's data stack: - Check for pipeline tools: `dags/` (Airflow), `dagster_home/`, `prefect.yaml`, `dbt_project.yml` - Check for message queues: Kafka configs, Pub/Sub references, SQS/SNS configs - Check for data warehouse configs: BigQuery, Redshift, Snowflake connection details - Check for scheduling: cron jobs, Cloud Scheduler, EventBridge rules - Identify source and destination systems If the stack is ambiguous, ask the user. ### Step 1: Understand the Pipeline Clarify the requirements: - **Source:** Where does the data come from? (API, database, file, stream) - **Destination:** Where does it need to go? (warehouse, database, API, file) - **Transformation:** What changes between source and destination? - **Schedule:** How often? Real-time, hourly, daily, on-demand? - **Volume:** How much data per run? Growth expectations? ### Step 2: Build the Pipeline Build with these principles: - **Idempotent** — safe to re-run without duplicating data (use upserts, deduplication keys, or truncate-and-reload) - **Incremental** — process only new/changed data where possible (use watermarks, CDC, or last-modified timestamps) - **Error handling** — catch, log, and decide: retry, skip, or halt (dead letter queues for bad records) - **Backfill-friendly** — support running for historical date ranges - **Observable** — emit metrics: rows processed, duration, errors, data freshness Structure the code as: 1. **Extract** — pull data from source with pagination, rate limiting, retries 2. **Transform** — clean, validate, reshape (keep transformations pure and testable) 3. **Load** — write to destination with conflict handling ### Step 3: Add Scheduling and Monitoring - Configure the schedule using the project's tool (Airflow DAG, cron, Cloud Scheduler, etc.) - Add monitoring hooks: alerting on failure, SLA tracking, data freshness checks - Include a health check endpoint or status query ### Step 4: Present the Pipeline ``` ## Pipeline Summary **Source:** [source] | **Destination:** [destination] | **Schedule:** [frequency] ### Data Flow source → extract → transform → load → destination ### Error Handling - [strategy for transient errors] - [strategy for bad records] ### Monitoring - [what is monitored] - [alerting thresholds] ### Backfill Run with: [command to backfill a date range] ``` ## Delivery If output exceeds the 40-line CLI budget, invoke `/atlas-report` with the full findings. The HTML report is the output. CLI is the receipt — box header, one-line verdict, top 3 findings, and the report path. Never dump analysis to CLI. --- Source attribution: tonone-ai Original project: https://github.com/tonone-ai/tonone/blob/1ceec80367a4e846e2f209cc87eea9d7544bd978/skills/flux-pipeline/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License Copyright (c) 2026 tonone-ai Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: tonone-ai Original project: https://github.com/tonone-ai/tonone/blob/1ceec80367a4e846e2f209cc87eea9d7544bd978/skills/flux-pipeline/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering, Operations, Product Management Roles: Business Process Analyst, Data Analyst, Data Engineer, ML / AI Engineer, Product Manager Specialties: API, Backend, Database Tools: Airflow, Bigquery, Dagster, Prefect, Redshift, Snowflake Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: a15bf03a6dc02d0e8d7a63ae599638460b520039944a1fb1d8d1588adb198be3