Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Thu, 28 Apr 2016 15:41:12 +0000 (UTC)
From: "Jeremiah Lowin (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12963495.1461852833000.60288.1461858072892@Atlassian.JIRA>
In-Reply-To: <JIRA.12963495.1461852833000@Atlassian.JIRA>
References: <JIRA.12963495.1461852833000@Atlassian.JIRA>
 <JIRA.12963495.1461852833312@arcas>
Subject: [jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/AIRFLOW-14?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1526=
2357#comment-15262357 ]=20

Jeremiah Lowin commented on AIRFLOW-14:
---------------------------------------

Yes, that situation could arise... I'm not sure of the best way to handle t=
his in a database-agnostic way (i.e.  SQLAlchemy). I'd appreciate any sugge=
stions!

> DagRun Refactor (Scheduler 2.0)
> -------------------------------
>
>                 Key: AIRFLOW-14
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-14
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Jeremiah Lowin
>            Assignee: Jeremiah Lowin
>              Labels: backfill, dagrun, scheduler
>
> For full proposal, please see the Wiki: https://cwiki.apache.org/confluen=
ce/pages/viewpage.action?pageId=3D62694286
> Borrowing from that page:=20
> *Description of New Workflow*
> DagRuns represent the state of a DAG at a certain point in time (perhaps =
they should be called DagInstances?). To run a DAG =E2=80=93 or to manage t=
he execution of a DAG =E2=80=93 a DagRun must first be created. This can be=
 done manually (simply by creating a DagRun object) or automatically, using=
 methods like dag.schedule_dag(). Therefore, both scheduling new runs OR in=
troducing ad-hoc runs can be done by any process at any time, simply by cre=
ating the appropriate object.
> Just creating a DagRun is not enough to actually run the DAG (just as cre=
ating a TaskInstance is not the same as actually running a task). We need a=
 Job for that. The DagRunJob is fairly simple in structure. It maintains a =
set of DagRuns that it is tasked with executing, and loops over that set un=
til all the DagRuns either succeed or fail. New DagRuns can be passed to th=
e job explicitly via DagRunJob.submit_dagruns() or by defining its DagRunJo=
b.collect_dagruns() method, which is called during each loop. When the DagR=
unJob is executing a specific DagRun, it locks it. Other DagRunJobs will no=
t try to execute locked DagRuns. This way, many DagRunJobs can run simultan=
eously in either a local or distributed setting, and can even be pointed at=
 the same DagRuns, without worrying about collisions or interference.
> The basic DagRunJob loop works like this:
> - refresh dags
> - collect new dagruns
> - process dagruns (including updating dagrun states for success/failure)
> - call executor/own heartbeat
> By tweaking the DagRunJob, we can easily recreate the behavior of the cur=
rent SchedulerJob and BackfillJob. The Scheduler simply runs forever and pi=
cks up ALL active DagRuns in collect_dagruns(); Backfill generates DagRuns =
corresponding to the requested start/end dates and submits them to itself p=
rior to initiating its loop.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)