airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyril Scetbon <cyril.scet...@free.fr>
Subject Re: Handling running tasks
Date Sun, 10 Jul 2016 16:02:11 GMT
Interesting. Thanks for this solution Lance, gonna try it
> On Jul 6, 2016, at 19:11, Lance Norskog <lance.norskog@gmail.com> wrote:
> 
> You could use the XCOM feature to post a semaphore at the start of the task
> and then remove it at the end.
> Another task would see the semaphore and immediately quit.
> If you get a race condition, the third 15-minute task will take care of the
> problem, 15 minutes late.
> 
> Lance
> 
> On Wed, Jul 6, 2016 at 2:50 PM, Cyril Scetbon <cyril.scetbon@free.fr> wrote:
> 
>> Hi,
>> 
>> I have dags with tasks that use same configuration and same schedule time
>> frequencies.
>> 
>> When I have x tasks in a dag they run in //. My dag is scheduled every 15
>> minutes, but sometimes I have at least a task that runs for more than 15
>> minutes and 2 identical tasks should not run at the same time (in my case)
>> but it's what happens.
>> 
>> So when the dag is kicked off (every 15 min) I need Airflow to run tasks
>> only for those that are not running, which means :
>> 
>> - if at time t1 tasks tk1 is running and tk2, ..., tkn are not running, I
>> need Airflow to run only tk2,.., tkn but not tk1 cause it's already running
>> - if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to run
>> tk1,.., tkn
>> 
>> I already tried using depends_on_past=True, however I need failed tasks to
>> be kicked off, cause I can get some temporary issues.
>> 
>> I also use an upstream task connected to all tasks in my dag to be able to
>> run manually all tasks if I need to. (I don't use a frequency of 15 minutes
>> in all dags)
>> 
>> Any idea ?
> 
> 
> 
> 
> -- 
> Lance Norskog
> lance.norskog@gmail.com
> Redwood City, CA


Mime
View raw message