Discovering Dagster orchestrator and how to deploy workflows on Kubernetes

My weekend immersion in Dagster, from pipeline creation to Kubernetes set up, what I learned, what I loved, what I hated

Stefano Bosisio
14 min readNov 24, 2023

Table of content

Install and spin up Dagster locally
Install Dagster on Kubernetes
— — Define a folder structure
— — Specify assets
— — Define partitions to read your input data
— — Link two assets together
— — Create a pipeline definition
— — Docker to build the Dagster pipeline
— — Create configmap in Kubernetes to host assets source codes
— — Set up the Helm values file
— — (Optional) Prepare your pipeline in parallel
— — Deploy your pipeline
Final remarks
Support my writing

In the ever-evolving world of data, where the demand for agility, reliability, and scalability is relentless, a lot of data and ML orchestrators emerge as possible conductors for your end-to-end ML cycle. Just to name a few we can have…

--

--

Stefano Bosisio

Machine Learning Engineer, PhD in Computational Chemistry. My writing covers neuroscience research, coding tutorial and social-media analyses