Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
References: <CADrzvVi4EgVyWPDycw9FuqKqN3Xhn8Lv-yYi1F56f6JfYKGsjQ@mail.gmail.com>
 <CAF=bis9u9FuPe0e-uqOW18LbFzKnT_Kkv11PX85iRnPetV5LeA@mail.gmail.com> <CAHEEp7XK8_rLgAM3MQDge6+_vF2nZ-8aR2ucdcHOsQyWr-XhgA@mail.gmail.com>
In-Reply-To: <CAHEEp7XK8_rLgAM3MQDge6+_vF2nZ-8aR2ucdcHOsQyWr-XhgA@mail.gmail.com>
From: Ben Tallman <ben@apigee.com>
Date: Tue, 21 Jun 2016 23:53:21 +0000
Message-ID: <CAF=bis9bGLojWdsy3h4YK-OfxCE1Lo=iVr8w-imQ8Uj=aMJbfQ@mail.gmail.com>
Subject: Re: dagrun_timeout not honoured?
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a114e522e2985ea0535d28709
archived-at: Tue, 21 Jun 2016 23:53:44 -0000

--001a114e522e2985ea0535d28709
Content-Type: text/plain; charset=UTF-8

Maxime -

Wish I did have time... BUT, I can say that SLA timeouts will fail (and
error) on any dag with schedule=None because they depend on the Now() being
less than the next scheduled date (None)...

dttm = dag.following_schedule(dttm)
while dttm < datetime.now():

I will file a bug...


On Tue, Jun 21, 2016 at 11:26 AM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> A tangent here: for people who have the knowledge (and a bit of time on
> their hand), providing a failing unit test can help the core committers
> with an easy way to jump in to help.
>
> I always wonder whether I'll be able reproduce the bug, is it version
> specific? environment specific? is it based on bad assumptions?
>
> With a failing unit test it's really clear what the expectations are and it
> makes it really easy for people can just jump in and fix it.
>
> Thanks,
>
> Max
>
> On Tue, Jun 21, 2016 at 9:15 AM, Ben Tallman <ben@apigee.com> wrote:
>
> > We have seen this too. Running 1.7.0 with Celery, neither DAG timeout nor
> > individual task sla's seem to be honored. In truth, we haven't done a lot
> > of testing, as it is more important that we get our overall ETL migrated
> > with workarounds.
> >
> > However, we will be digging in at some point for greater clarity...
> >
> > On Thu, Jun 16, 2016 at 11:21 AM harish singh <harish.singh22@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > Since we have "dag_conurrency" restriction, I tried to play with
> > >  dagrun_timeout.
> > > So that after some interval, dag runs are marked failed and pipeline
> > > progresses.
> > > But this is not happening.
> > >
> > > I have this dag (@hourly):
> > >
> > > A -> B -> C  -> D -> E
> > >
> > > C: depends_on_past=true
> > >
> > > My dagrun_timeout is 60 minutes
> > >
> > > default_args = {
> > >     'owner': 'airflow',
> > >     'depends_on_past': False,
> > >     'start_date': scheduling_start_date,
> > >     'email': ['airflow@airflow.com'],
> > >     'email_on_failure': False,
> > >     'email_on_retry': False,
> > >     'retries': 2,
> > >     'retry_delay': default_retries_delay,
> > >     'dagrun_timeout':datetime.timedelta(minutes=60)
> > > }
> > >
> > >
> > > Parallelism setting in airflow.cfg:
> > >
> > > parallelism = 8
> > > dag_concurrency = 8
> > > max_active_runs_per_dag = 8
> > >
> > >
> > > For hour 1, all the tasks got completed.
> > > Now in hour 2,  say task C failed.
> > >
> > > From hour 3 onwards, Tasks A and B keep running.
> > > Task C never triggers because it depends on past (and past hour failed)
> > >
> > > Since dag conurrency is 8, my pipeline progresses from hour 3 to hour
> 10
> > > (thats next 8 hours) for Tasks A and B. After this, pipeline stalls.
> > >
> > > "dagrun_timeout" was 60 minutes.  This should mean that after 60
> minutes,
> > > from hour 3 onwards, the DAG runs that has been up for more than 60
> > minutes
> > > should be marked FAILED  and the pipeline should progress?
> > >
> > > But this is not happening. So I am guessing my understanding here is
> not
> > > correct.
> > >
> > > What should be behavior when we use  "dagrun_timeout" ?
> > > Also, how can I make sure that the dag proceeds in this situation?
> > >
> > > In the example I gave above, Task A and B should keep running every
> hour
> > > (since it doesnt depend on past).
> > > Why it runs 8(dag_conurrency) instances  and stalls?
> > >
> > >
> > > Thanks,
> > > Harish
> > >
> >
>

--001a114e522e2985ea0535d28709--