airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Alpedrinha Ramos <jalpedrinhara...@gmail.com>
Subject Re: Airflow Newbie Question
Date Wed, 22 Mar 2017 17:06:45 GMT
Defining the dag inside the for loop would be the best way IMO, if you
define a subdag you would have the same problem.

Subdags seem to be more suited for repeating some patters without rewriting
the code.


On Wed, Mar 22, 2017 at 4:49 PM Andrew Maguire <andrewm4894@gmail.com>
wrote:

> Yeah was thinking that afterwards. Would that be a sub dag approach, or is
> it even as  easy as moving the dag definition itself into the for loop?
>
> Cheers
> Andy
>
> On Wed, Mar 22, 2017 at 4:45 PM Jorge Alpedrinha Ramos <
> jalpedrinharamos@gmail.com> wrote:
>
> > Hey!
> >
> > I've read your post and I would point that you could instead of creating
> > tasks on one dag for each lob, you could create one dag per each lob
> with a
> > 'my_bigquery_dag_for_{{lob}}'. This way you get a more isolated unit of
> > management (dag). The advantage here being that in the event of a failure
> > or data reprocess for a single lob you wouldn't need to reprocess all
> lobs.
> >
> > Other than that it is really cool that you're putting it out on a blog
> > post, it will definitely be useful.
> >
> > Cheers !
> >
> > On Wed, Mar 22, 2017 at 4:20 PM Andrew Maguire <andrewm4894@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Just to follow up as fyi.
> > >
> > > Think i figured out at least one approach. Did a blog post on it
> > >
> > >
> >
> http://engineering.pmc.com/2017/03/playing-around-with-apache-airflow-bigquery-62/
> > >
> > > Can't wait to get going on porting stuff over from cron.
> > >
> > > Cheers,
> > > Andy
> > >
> > > On Tue, Mar 21, 2017 at 2:49 PM Andrew Maguire <andrewm4894@gmail.com>
> > > wrote:
> > >
> > > > Sorry if this is not the right place for this.
> > > >
> > > > I'm wondering if anyone might be able to help me with this question?
> > > >
> > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/42930221/best-way-to-loop-through-parameters-in-airflow
> > > >
> > > > Feel free to ignore if this is not the right place to try find help.
> > > >
> > > > Loving airflow btw!
> > > >
> > > > Cheers,
> > > > Andy
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message