airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: AIRFLOW-699 - dags can't be triggered at the same second
Date Sun, 25 Dec 2016 12:05:25 GMT
Hi,

If you are running on Postgres and 1.7.1.3 or Mysql/MariaDB and master triggering a dag run
within the same second should work as long they are not triggered at the exact same time (6
fractions of a second).

Robin makes the correct case to make this work for exact same time triggers it would require
rewiring task instances to use the primary key of dag runs. I’ve done this is in the past
but received some push back on this from Jeremiah and thus removed it from the PR at that
time.

Bolke.

> Op 21 dec. 2016, om 10:58 heeft Miller, Robin <Robin.Miller@affiliate.oliverwyman.com>
het volgende geschreven:
> 
> Hi Olivier,
> 
> 
> Someone correct me if I'm wrong, but my understanding is as follows:
> 
> 
> At the moment the only way to identify a task_instance as for a specific dag_run is by
comparing the dag_id and execution_date. As such, having two dag_runs with the same execution_date
would make it impossible to correctly identify the task states for each run. So that constraint
is important.
> 
> 
> In order to remove this constraint, I believe Airflow would need to assign a unique key
to a dag_run (possibly per dag_id, but that might be harder to implement) and use that key
to identify its task instances instead of the execution date.
> 
> 
> Regards,
> 
> Robin Miller
> OLIVER WYMAN
> robin.miller@affiliate.oliverwyman.com<mailto:robin.miller@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
> 
> ________________________________
> From: Olivier Girardot <o.girardot@lateral-thoughts.com>
> Sent: 21 December 2016 07:02:05
> To: dev@airflow.incubator.apache.org
> Subject: Re: AIRFLOW-699 - dags can't be triggered at the same second
> 
> anyone ?
> 
> 
> 
> 
> 
> 
> On Fri, Dec 16, 2016 2:46 PM, Olivier Girardot o.girardot@lateral-thoughts.com
> wrote:
> Hi everyone,I wanted to talk about an issue I created AIRFLOW-699, there's an
> integrity constraint regarding the dag_id and the execution_date that makes it
> impossible to trigger two times with different parameters a specific dag at the
> same second.
> As we are using airflow as the scheduler at the end of a complex notification
> process, during high load scenario, these things happen.
> Is there any hard reason for this uniqueness constraint ?
> Regards,
> Olivier Girardot
> 
> 
> Olivier Girardot| Associé
> o.girardot@lateral-thoughts.com
> +33 6 24 09 17 94
> 
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged. If you received
this message in error or are not the intended recipient, you should destroy the e-mail message
and any attachments or copies, and you are prohibited from retaining, distributing, disclosing
or using any information contained herein. Please inform us of the erroneous delivery by return
e-mail. Thank you for your cooperation.


Mime
View raw message