Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Mon, 9 May 2016 18:51:12 +0000 (UTC)
From: "Chris Riccomini (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12963495.1461852833000.145041.1462819872903@Atlassian.JIRA>
In-Reply-To: <JIRA.12963495.1461852833000@Atlassian.JIRA>
References: <JIRA.12963495.1461852833000@Atlassian.JIRA> <JIRA.12963495.1461852833312@arcas>
Subject: [jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 09 May 2016 18:51:17 -0000


    [ https://issues.apache.org/jira/browse/AIRFLOW-14?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1527=
6805#comment-15276805 ]=20

Chris Riccomini commented on AIRFLOW-14:
----------------------------------------

Yea, I agree with this. I guess the two things that I'm worried about are:

# Will this mean we can't safely run more than one LocalExecutor?
# Will this break the CeleryExecutor?

> DagRun Refactor (Scheduler 2.0)
> -------------------------------
>
>                 Key: AIRFLOW-14
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-14
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Jeremiah Lowin
>            Assignee: Jeremiah Lowin
>              Labels: backfill, dagrun, scheduler
>
> For full proposal, please see the Wiki: https://cwiki.apache.org/confluen=
ce/pages/viewpage.action?pageId=3D62694286
> Borrowing from that page:=20
> *Description of New Workflow*
> DagRuns represent the state of a DAG at a certain point in time (perhaps =
they should be called DagInstances?). To run a DAG =E2=80=93 or to manage t=
he execution of a DAG =E2=80=93 a DagRun must first be created. This can be=
 done manually (simply by creating a DagRun object) or automatically, using=
 methods like dag.schedule_dag(). Therefore, both scheduling new runs OR in=
troducing ad-hoc runs can be done by any process at any time, simply by cre=
ating the appropriate object.
> Just creating a DagRun is not enough to actually run the DAG (just as cre=
ating a TaskInstance is not the same as actually running a task). We need a=
 Job for that. The DagRunJob is fairly simple in structure. It maintains a =
set of DagRuns that it is tasked with executing, and loops over that set un=
til all the DagRuns either succeed or fail. New DagRuns can be passed to th=
e job explicitly via DagRunJob.submit_dagruns() or by defining its DagRunJo=
b.collect_dagruns() method, which is called during each loop. When the DagR=
unJob is executing a specific DagRun, it locks it. Other DagRunJobs will no=
t try to execute locked DagRuns. This way, many DagRunJobs can run simultan=
eously in either a local or distributed setting, and can even be pointed at=
 the same DagRuns, without worrying about collisions or interference.
> The basic DagRunJob loop works like this:
> - refresh dags
> - collect new dagruns
> - process dagruns (including updating dagrun states for success/failure)
> - call executor/own heartbeat
> By tweaking the DagRunJob, we can easily recreate the behavior of the cur=
rent SchedulerJob and BackfillJob. The Scheduler simply runs forever and pi=
cks up ALL active DagRuns in collect_dagruns(); Backfill generates DagRuns =
corresponding to the requested start/end dates and submits them to itself p=
rior to initiating its loop.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)