airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deng Xiaodong <xd.den...@gmail.com>
Subject Re: Table "dag_run" has unique constraint on (dag_id, execution_date)
Date Fri, 18 Jan 2019 23:37:40 GMT
Without this unique constraint, how is the scheduler supposed to
find/update the state of each specific TaskInstance or DagRun? How should
the log be named & stored?

It should be there.


XD

On Sat, Jan 19, 2019 at 03:20 Ash Berlin-Taylor <ash@apache.org> wrote:

> So that it's here on a searchable record:
>
> Simply this is how airflow identifies tasks - `airflow run my_dag_id
> 2018-01-17 my task`.
>
> If you were to have two dag runs for the exact same milliseconds which one
> would it run? Much of the system treats this combo as unique, so lots of
> code would need to change to make it not.
>
> But: what is your use case here? The same dag running twice at for the
> same period doesn't make sense to me conceptual (but then I admit I now
> think in terms of what airflow does internally do my thinking is coloured)
>
> -ash
>
> On 18 January 2019 19:14:01 GMT, "Andreas K├Âltringer" <
> andreas.koeltringer@n-fuse.co> wrote:
> >Hi,
> >
> >almost a year ago I reported an issue to Airflow's Jira:
> >https://issues.apache.org/jira/browse/AIRFLOW-2319
> >
> >Recently, someone pointed out that I should ask here on the mailing
> >list (I
> >thought I did, but apparently, I did not).
> >
> >So here's the thing: the DagRun table has a unique constraint on
> >(dag_id,
> >exec_date), that was not reflected by models.py (now
> >models/__init__.py). This
> >got "fixed" in June 2018 - apparently this was not the only place where
> >model
> >declaration and schema migration scripts were out of sync.
> >
> >Ash now pointed out on AIRFLOW-2319, that the unique constraint is
> >there "by
> >design", meaning there's a reason why it's there. Can somebody please
> >explain
> >to me what's the reasoning behind this? For us it would be a use case
> >to have
> >multiple dag runs per (dag_id, exec_date).
> >
> >thx in advance!
> >
> >--
> >Andreas Koeltringer
> >Mail:   andreas.koeltringer@n-fuse.co
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message