airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Toonstra <gtoons...@gmail.com>
Subject Re: Airflow state change diagram
Date Sun, 20 Nov 2016 22:16:02 GMT
Hi Li,

I used graphviz (the dot app) to generate it, but the data was collected
manually.

Rgds,

Gerard


On Sat, Nov 19, 2016 at 11:42 PM, Li Xuan Ji <xuanji@gmail.com> wrote:

> Thanks Gerard!
>
> Did you generate this manually?
>
> On 2 November 2016 at 11:07, Joseph Napolitano
> <joseph.napolitano@blueapron.com.invalid> wrote:
> > Gerard,
> >
> > This is great.  Thanks for sharing this.
> >
> > Joe
> >
> > On Wed, Nov 2, 2016 at 7:43 AM, twinkle sachdeva <
> twinkle.sachdeva@gmail.com
> >> wrote:
> >
> >> Thanks Gerard for sharing it.
> >>
> >> Regards,
> >> Twinkle
> >>
> >> On Mon, Oct 31, 2016 at 2:35 AM, Gerard Toonstra <gtoonstra@gmail.com>
> >> wrote:
> >>
> >> > I was looking at trying to fix AIRFLOW-137 (max_active_runs not
> >> respected),
> >> > but quickly noticed that the code that does all the scheduling is
> rather
> >> > complex with state updates going on across multiple source files in
> >> > multiple threads, etc.
> >> >
> >> > It's then best to find a suitable way to visualize all this
> complexity,
> >> so
> >> > I built this state change diagram:
> >> >
> >> > https://docs.google.com/spreadsheets/d/1vVvOwfDSacTC_
> >> YzwUkOMyykP6LiipCeoW_
> >> > V70PuFrN4/edit?usp=sharing
> >> >
> >> > The state changes represent a potential execution path where the state
> >> for
> >> > a task instance will be updated to that value. Backfill is not
> considered
> >> > in this diagram. States for dagruns/jobs/dags are also not considered.
> >> >
> >> > Could be useful for someone else.
> >> >
> >> > Rgds,
> >> >
> >> > Gerard
> >> >
> >>
> >
> >
> >
> > --
> > *Joe Napolitano *| Sr. Data Engineer
> > www.blueapron.com | 5 Crosby Street, New York, NY 10013
>
>
>
> --
> Im Xuan Ji!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message