airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J C Lawrence <c...@kanga.nu>
Subject Re: Dynamically defining tasks in a DAG -- HOW?
Date Thu, 08 Sep 2016 23:12:38 GMT
On Thu, 8 Sep 2016 14:04:58 -0700, Ben Tallman <ben@apigee.com> wrote:

> We have done this a lot, and the one issue is that every time the DAG
> is evaluated (even during a run), the SQL will be re-run, and tasks
> can vary. In fact, we had a select statement that actually marked
> items as in process during select, and THAT was bad.

Yeah, I'm keeping an eye on that.

The problem I'm having however is that the DAGs are not getting
populated with the tasks relevant to that specific scheduling run.  Do
you have this working under Airflow today?

> We have moved to x number of tasks, and each one grabs a line from
> the DB, and 0 to n of them can actually get skipped if they don't get
> a line from the DB.

Yeah, I just don't want to a) setup yet another DB/table for an
interstitial process or b) to re-invent dispatch/locking against a DB
yet again.

-- JCL

Mime
View raw message