airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Napolitano <joseph.napolit...@blueapron.com.INVALID>
Subject Re: Airflow state change diagram
Date Wed, 02 Nov 2016 15:07:15 GMT
Gerard,

This is great.  Thanks for sharing this.

Joe

On Wed, Nov 2, 2016 at 7:43 AM, twinkle sachdeva <twinkle.sachdeva@gmail.com
> wrote:

> Thanks Gerard for sharing it.
>
> Regards,
> Twinkle
>
> On Mon, Oct 31, 2016 at 2:35 AM, Gerard Toonstra <gtoonstra@gmail.com>
> wrote:
>
> > I was looking at trying to fix AIRFLOW-137 (max_active_runs not
> respected),
> > but quickly noticed that the code that does all the scheduling is rather
> > complex with state updates going on across multiple source files in
> > multiple threads, etc.
> >
> > It's then best to find a suitable way to visualize all this complexity,
> so
> > I built this state change diagram:
> >
> > https://docs.google.com/spreadsheets/d/1vVvOwfDSacTC_
> YzwUkOMyykP6LiipCeoW_
> > V70PuFrN4/edit?usp=sharing
> >
> > The state changes represent a potential execution path where the state
> for
> > a task instance will be updated to that value. Backfill is not considered
> > in this diagram. States for dagruns/jobs/dags are also not considered.
> >
> > Could be useful for someone else.
> >
> > Rgds,
> >
> > Gerard
> >
>



-- 
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message