Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <F04705EC-9F3E-45BE-B55F-D944873C39E0@free.fr>
References: <BC3A7F53-C67B-4EB1-913B-1CC6E967DC68@free.fr> <CAP0ykh_OcEza_A6gN+kaPwb_F3RWKevgSjpa_ybWoRBE87FCag@mail.gmail.com>
 <F04705EC-9F3E-45BE-B55F-D944873C39E0@free.fr>
From: siddharth anand <sanand@apache.org>
Date: Tue, 2 Aug 2016 23:37:00 -0700
Message-ID: <CANLtMideTF-6JjH0BasZ5xWf3LHbcg-iB9WFpD9-QD88sxR2Zw@mail.gmail.com>
Subject: Re: Handling running tasks
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113ce6c29393d40539250f60
archived-at: Wed, 03 Aug 2016 06:37:07 -0000

--001a113ce6c29393d40539250f60
Content-Type: text/plain; charset=UTF-8

+1 on the uniqueness of the solution. Wondering how it worked!
-s


On Sun, Jul 10, 2016 at 9:02 AM, Cyril Scetbon <cyril.scetbon@free.fr>
wrote:

> Interesting. Thanks for this solution Lance, gonna try it
> > On Jul 6, 2016, at 19:11, Lance Norskog <lance.norskog@gmail.com> wrote:
> >
> > You could use the XCOM feature to post a semaphore at the start of the
> task
> > and then remove it at the end.
> > Another task would see the semaphore and immediately quit.
> > If you get a race condition, the third 15-minute task will take care of
> the
> > problem, 15 minutes late.
> >
> > Lance
> >
> > On Wed, Jul 6, 2016 at 2:50 PM, Cyril Scetbon <cyril.scetbon@free.fr>
> wrote:
> >
> >> Hi,
> >>
> >> I have dags with tasks that use same configuration and same schedule
> time
> >> frequencies.
> >>
> >> When I have x tasks in a dag they run in //. My dag is scheduled every
> 15
> >> minutes, but sometimes I have at least a task that runs for more than 15
> >> minutes and 2 identical tasks should not run at the same time (in my
> case)
> >> but it's what happens.
> >>
> >> So when the dag is kicked off (every 15 min) I need Airflow to run tasks
> >> only for those that are not running, which means :
> >>
> >> - if at time t1 tasks tk1 is running and tk2, ..., tkn are not running,
> I
> >> need Airflow to run only tk2,.., tkn but not tk1 cause it's already
> running
> >> - if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to
> run
> >> tk1,.., tkn
> >>
> >> I already tried using depends_on_past=True, however I need failed tasks
> to
> >> be kicked off, cause I can get some temporary issues.
> >>
> >> I also use an upstream task connected to all tasks in my dag to be able
> to
> >> run manually all tasks if I need to. (I don't use a frequency of 15
> minutes
> >> in all dags)
> >>
> >> Any idea ?
> >
> >
> >
> >
> > --
> > Lance Norskog
> > lance.norskog@gmail.com
> > Redwood City, CA
>
>

--001a113ce6c29393d40539250f60--