Discovering Dagster orchestrator and how to deploy workflows on Kubernetes

My weekend immersion in Dagster, from pipeline creation to Kubernetes set up, what I learned, what I loved, what I hated

Image by Christian Kapeller on Unsplash

Table of content

— Install and spin up Dagster locally
— Install Dagster on Kubernetes
— — Define a folder structure
— — Specify assets
— — Define partitions to read your input data
— — Link two assets together
— — Create a pipeline definition
— — Docker to build the Dagster pipeline
— — Create configmap in Kubernetes to host assets source codes
— — Set up the Helm values file
— — (Optional) Prepare your pipeline in parallel
— — Deploy your pipeline
— Final remarks
— Support my writing

In the ever-evolving world of data, where the demand for agility, reliability, and scalability is relentless, a lot of data and ML orchestrators emerge as possible conductors for your end-to-end ML cycle. Just to name a few we can have…