airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From harish singh <harish.sing...@gmail.com>
Subject Re: Performance: backfill --mark_success
Date Tue, 06 Dec 2016 21:44:14 GMT
I am working around this in a not-so-pretty-hacky solution.
Instead of "backfill -m" for the dag I am using the "-t"   flag and marking
success only the 1st Task of my pipeline.  Once the backfill is complete, I
used the UI to "Mark success" all "Future" and "Downstream" tasks.

Max,
I am not sure I clearly understood about individual "heartrate" per job.
Can "job_heartbeat_sec" specified for per Task basis?

Thanks,
Harish

On Tue, Dec 6, 2016 at 9:31 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> Oh thanks for pointing this out, I just did a round of review on that PR.
>
> While we have people's attention around backfill on this thread, I'd love
> to introduce the new term "scheduler catchup" as something distinct to
> `backfill`, at least until we get single code path for both operations.
>
> Max
>
> On Tue, Dec 6, 2016 at 8:56 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
>> There is pr out for not having backfills at all specified at the dag
>> level as well.
>>
>>
>>
>> *Van: *Maxime Beauchemin <maximebeauchemin@gmail.com>
>> *Verzonden: *dinsdag 6 december 2016 16:54
>> *Aan: *dev@airflow.incubator.apache.org
>> *CC: *harish.singh22@gmail.com
>> *Onderwerp: *Re: Performance: backfill --mark_success
>>
>>
>>
>> The backfill `mark_success` logic could really be optimized by not relying
>>
>> on `airflow run --mark_success` by altering the database state directly
>>
>> instead of actually triggering tasks at all and relying on the backfill
>>
>> logic. Simply scope the set of task instances in scope, and merge (upsert)
>>
>> a `success` state to the db directly.
>>
>>
>>
>> To accelerate it though as it is today, you can reduce some of the
>>
>> heartbeats configurations (job_heartbeat_sec). It's usually desirable to
>>
>> have this setting lower in dev (say 5 seconds) than in production (30-60
>>
>> seconds).
>>
>>
>>
>> I suggest that better default that would be individually configurable for
>>
>> `heartrate` be in place for different types of jobs in `jobs.py`.
>>
>>
>>
>> Max
>>
>>
>>
>> On Mon, Dec 5, 2016 at 12:39 PM, Laura Lorenz <llorenz@industrydive.com>
>>
>> wrote:
>>
>>
>>
>> > This is not that helpful of a message, but I also was having a problem
>> with
>>
>> > `airflow backfill -m` on Airflow version 1.7.0 with it going super
>> slow. In
>>
>> > the end I got around the necessity in that specific case, thinking that
>> it
>>
>> > was broken in 1.7.0 re (
>>
>> > https://gitter.im/apache/incubator-airflow?at=56c3956c1f9833ef7c9ba8ee
>> ),
>>
>> > but now that I am writing this and triangulating 1.7.0's release and
>> that
>>
>> > gitter comment, it doesn't make sense. I'll give it another go.
>>
>> >
>>
>> > On Sun, Dec 4, 2016 at 7:50 AM, Vikas Malhotra <
>> vikas.malhotra08@gmail.com
>>
>> > >
>>
>> > wrote:
>>
>> >
>>
>> > > Hello Harish,
>>
>> > >
>>
>> > > Based on our understanding of Python Multiprocessing, a task instance
>>
>> > gets
>>
>> > > a record in underlying database after there is an explicit call to
>>
>> > airflow
>>
>> > > from that library (using Local Executor). So, I might be wrong, but
>> you
>>
>> > > won't find a record in database until and unless that task instance
>> has
>>
>> > got
>>
>> > > initiated. I might be wrong in our assumptions and would love to be
>>
>> > > corrected if that's the case.
>>
>> > >
>>
>> > > We have been using latest only operator and it's seems to be working
>> well
>>
>> > > for skipping tasks if they are not current (basically avoiding
>> backfill
>>
>> > by
>>
>> > > marking all tasks below the latest only operator as skipped). It's
>>
>> > present
>>
>> > > in master branch as of now and I would recommend you to look at that
>>
>> > > operator for backfill.
>>
>> > >
>>
>> > > Thanks!
>>
>> > > Vikas
>>
>> > >
>>
>> > > On Dec 4, 2016 5:23 AM, "harish singh" <harish.singh22@gmail.com>
>> wrote:
>>
>> > >
>>
>> > > Hi all,
>>
>> > >
>>
>> > > We have been running Airflow in our production for over 8-9 months
>> now.
>>
>> > > I know there is a separate thread in place for Airflow 2.0.
>>
>> > > But I was not sure if any of the prior version has this fixed.  If
>> not, I
>>
>> > > will add this to the other email thread for 2.0.
>>
>> > >
>>
>> > > When I run airflow backfill with "-m"  (Mark jobs as succeeded without
>>
>> > > running them) ,
>>
>> > > is there a way to optimize this call?
>>
>> > >
>>
>> > > For example:
>>
>> > > airflow backfill TEST_DAG -s 2016-11-01T00:00:00 -e
>> 2016-12-01T00:00:00
>>
>> > -m
>>
>> > >
>>
>> > > Here, I am running backfill for a month (from 1st Nov to 1st Dec).
>>
>> > > Essentially, Marking the jobs as succeeded without running them.
>>
>> > >
>>
>> > > It has ben more than an hour and the backfill has managed to reach
>> only
>>
>> > > upto 2nd Nov.
>>
>> > > This seems to be very slow when there is no need to even run the
>> tasks.
>>
>> > >
>>
>> > >
>>
>> > > I am running Airflow 1.7.0:
>>
>> > > These are my related configuration settings:
>>
>> > >
>>
>> > > parallelism = 50
>>
>> > > dag_concurrency = 20
>>
>> > > max_active_runs_per_dag = 8
>>
>> > >
>>
>> > > Also, I have around 9 Dags running (all Hourly). The other 8 dags are
>>
>> > > running as scheduled with start_date of 2016-11-01T00:00:00
>>
>> > >
>>
>> > > My question is, since I am only Marking the jobs as "succeeded"
>>
>> > > without running them,
>>
>> > > can this be done over 1 sql query, instead of per hour, per task
>> basis?
>>
>> > > May be find out all the TaskInstances that needs to be mark succeeded
>>
>> > > and then just run a sql?
>>
>> > >
>>
>> > > I may not be aware of lot of things here and very possible I am
>>
>> > > assuming a lot of things, incorrectly.
>>
>> > > Please feel free to correct me.
>>
>> > >
>>
>> > >
>>
>> > > Thanks,
>>
>> > > Harish
>>
>> > >
>>
>> >
>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message