Machine Learning Ops Ml Pipeline

# Machine Learning Pipeline - Multi-Agent MLOps Orchestration

Design and implement a complete ML pipeline for: $ARGUMENTS

Use this skill when

- Working on machine learning pipeline - multi-agent mlops orchestration tasks or workflows

- Needing guidance, best practices, or checklists for machine learning pipeline - multi-agent mlops orchestration

Do not use this skill when

- The task is unrelated to machine learning pipeline - multi-agent mlops orchestration

- You need a different domain or tool outside this scope

Instructions

- Clarify goals, constraints, and required inputs.

- Apply relevant best practices and validate outcomes.

- Provide actionable steps and verification.

- If detailed examples are required, open `resources/implementation-playbook.md`.

Thinking

This workflow orchestrates multiple specialized agents to build a production-ready ML pipeline following modern MLOps best practices. The approach emphasizes:

- **Phase-based coordination**: Each phase builds upon previous outputs, with clear handoffs between agents

- **Modern tooling integration**: MLflow/W&B for experiments, Feast/Tecton for features, KServe/Seldon for serving

- **Production-first mindset**: Every component designed for scale, monitoring, and reliability

- **Reproducibility**: Version control for data, models, and infrastructure

- **Continuous improvement**: Automated retraining, A/B testing, and drift detection

The multi-agent approach ensures each aspect is handled by domain experts:

- Data engineers handle ingestion and quality

- Data scientists design features and experiments

- ML engineers implement training pipelines

- MLOps engineers handle production deployment

- Observability engineers ensure monitoring

Phase 1: Data & Requirements Analysis

<Task>

subagent_type: data-engineer

prompt: |

Analyze and design data pipeline for ML system with requirements: $ARGUMENTS

Deliverables:

1. Data source audit and ingestion strategy:

- Source systems and connection patterns

- Schema validation using Pydantic/Great Expectations

- Data versioning with DVC or lakeFS

- Incremental loading and CDC strategies

2. Data quality framework:

- Profiling and statistics generation

- Anomaly detection rules

- Data lineage tracking

- Quality gates and SLAs

3. Storage architecture:

- Raw/processed/feature layers

- Partitioning strategy

- Retention policies

- Cost optimization

Provide implementation code for critical components and integration patterns.

</Task>

<Task>

subagent_type: data-scientist

prompt: |

Design feature engineering and model requirements for: $ARGUMENTS

Using data architecture from: {phase1.data-engineer.output}

Deliverables:

1. Feature engineering pipeline:

- Transformation specifications

- Feature store schema (Feast/Tecton)

- Statistical validation rules

- Handling strategies for missing data/outliers

2. Model requirements:

- Algorithm selection rationale

- Performance metrics and baselines

- Training data requirements

- Evaluation criteria and thresholds

3. Experiment design:

- Hypothesis and success metrics

- A/B testing methodology

- Sample size calculations

- Bias detection approach

Include feature transformation code and statistical validation logic.

</Task>

Phase 2: Model Development & Training

<Task>

subagent_type: ml-engineer

prompt: |

Implement training pipeline based on requirements: {phase1.data-scientist.output}

Using data pipeline: {phase1.data-engineer.output}

Build comprehensive training system:

1. Training pipeline implementation:

- Modular training code with clear interfaces

- Hyperparameter optimization (Optuna/Ray Tune)

- Distributed training support (Horovod/PyTorch DDP)

- Cross-validation and ensemble strategies

2. Experiment tracking setup:

- MLflow/Weights & Biases integration

- Metric logging and visua