---
name: fabric-pipeline-d6a5b1029b3a
description: Create, deploy, and test a Fabric Data Factory pipeline that chains all topic notebooks in order (download → bronze → dq_bronze → silver → dq_silver → gold → dq_gold). Use after all notebooks for a topic have been individually deployed and smoke-tested.
---

# fabric-pipeline

## MUST

- All notebooks for the topic must be deployed and individually smoke-tested before creating the pipeline
- Pipeline names follow the convention `pipeline_<topic>`
- Activities execute in strict order — each depends on its predecessor with `Succeeded`; failures never cascade
- For a **new** pipeline, always use `test` (create + run + monitor) — never report completion after `create` alone; activity wiring is only validated by a successful end-to-end run
- Use `create` alone only when the human explicitly says to skip the run (e.g. they will schedule the first run manually)
- Report STATUS: Completed or STATUS: Failed to orchestrator and stop — never re-run autonomously
- Each pipeline run consumes Fabric capacity; always await human approval before re-running a failed pipeline

## PREFER

- `test` command over separate `create` + `run` + `status` when running an existing pipeline
- Auto-discover over `--notebooks` — auto-discover reads naming conventions directly from the workspace
- Check the Fabric portal Run History for per-activity logs on failure

## AVOID

- Creating a pipeline before all its notebooks are deployed in Fabric
- Re-running on failure without human approval

## Commands

```bash
# 0. Confirm active workspace (every session, before any pipeline action)
fabric-vibe workspace switch list
# Stop. Ask: "Active workspace is '<displayName>'. Confirm to proceed?"
# Do not run pipeline manage until the human confirms.
# To deploy to a different workspace instead, use workspace transfer:
#   fabric-vibe workspace transfer --pipeline <topic> --to <displayName>

# Full cycle: auto-discover notebooks, create/update pipeline, run, and monitor
fabric-vibe pipeline manage test --topic lux_energy_price

# Create or update only (no run)
fabric-vibe pipeline manage create --topic lux_energy_price

# Explicit ordered list (use when auto-discover picks up the wrong notebooks)
fabric-vibe pipeline manage create --topic lux_energy_price \
    --notebooks download_lux_energy_price,bronze_lux_energy_price,dq_bronze_lux_energy_price

# Trigger an already-deployed pipeline (no monitoring)
fabric-vibe pipeline manage run --pipeline pipeline_lux_energy_price

# Monitor an existing run to completion
fabric-vibe pipeline manage status \
    --pipeline pipeline_lux_energy_price --instance <job-instance-id>

# List all data pipelines in the workspace
fabric-vibe pipeline manage list
```

## Activity Ordering

The tool sorts discovered notebooks by naming prefix. Use `--notebooks` to override.

| Prefix | Order | Layer responsibility |
|---|---|---|
| `download_` | 1 | Fetch source data to sandbox |
| `bronze_` | 2 | Ingest raw files to Bronze Delta |
| `dq_bronze_` | 3 | Validate Bronze layer |
| `silver_` | 4 | Clean and transform to Silver |
| `dq_silver_` | 5 | Validate Silver layer |
| `gold_` | 6 | Aggregate to Gold |
| `dq_gold_` | 7 | Validate Gold layer |

Notebooks that do not match a known prefix are appended at the end in alphabetical order.

## STATUS Handling

```
STATUS: Completed  → PASS — report to orchestrator
STATUS: Failed     → FAIL — report pipeline name, job instance ID, and failureReason.
                     STOP. Await human approval before any re-run.
STATUS: Cancelled  → report to orchestrator. Do not re-run without human approval.
```

For per-activity failure details: Fabric portal → Workspace → pipeline item → Run history → select the failed run.


---
Source attribution: scardoso-lu
Original project: https://github.com/scardoso-lu/fabric-skills-settings/blob/33534eb185fda573125308a0129457bd65419d13/server/skills/fabric-pipeline/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Dave AI.

## Upstream license notice

MIT License

Copyright (c) 2026 Silvio Cardoso

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Dave AI resource metadata

Creator / publisher: scardoso-lu
Original project: https://github.com/scardoso-lu/fabric-skills-settings/blob/33534eb185fda573125308a0129457bd65419d13/server/skills/fabric-pipeline/SKILL.md
Attribution basis: Source repository record

Fields of work: Development & Engineering, Product Management
Roles: Data Engineer, ML / AI Engineer, Product Manager
Specialties: API, Backend
Tools: Fabric
Country applicability: Country not specified
Classification basis: Source catalog labels
Instruction SHA-256: 3c30745abbcca69445d18965928a42dd1f7fd86c67030d05a8088cd6c0211c52