Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
Message-ID: <pony-b58493afe2ab0d043b6191d92bb74cd3f95ab95e-9b6e0095fedf063613d1f2b076ff34e93f3f6ce6@dev.airflow.apache.org>
Subject: Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours later
References: <CA+vv_nGX4pT3WrPcUemQZGCrhFuaAB5-=J13rnGAkgFOGc+dpw@mail.gmail.com> <pony-b58493afe2ab0d043b6191d92bb74cd3f95ab95e-e3ce155a5da135c83423bc0fcb5e10dc7fbe243c@dev.airflow.apache.org>
 <CA+vv_nFBKFBz8ADBKXboW5M1qAoAqP6r_7aW_D_cjvvpB2fV3w@mail.gmail.com> <pony-b58493afe2ab0d043b6191d92bb74cd3f95ab95e-517b818d2c46772f4680173b146d76d4503e911d@dev.airflow.apache.org>
From: "jspsai@gmail.com"<jspsai@gmail.com>
In-Reply-To: <CA+vv_nGX4pT3WrPcUemQZGCrhFuaAB5-=J13rnGAkgFOGc+dpw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Date: Wed, 04 Oct 2017 12:51:11 -0000
To: <dev@airflow.apache.org>
archived-at: Wed, 04 Oct 2017 19:37:29 -0000


On 2017-10-03 11:20, Justin Palmer <justin@github.com> wrote: 
> Hi Sai,
> 
> That operator would be a plugin
> <https://airflow.incubator.apache.org/plugins.html#plugins>, so you'll need
> to import it as such using plugins.operators.
> 
> from plugins.operators.execute_dag_run_operator import ExecuteDagRunOperator
> 
> 
> -Justin
> 
> On Tue, Oct 3, 2017 at 8:13 AM jspsai@gmail.com <jspsai@gmail.com> wrote:
> 
> >
> >
> > On 2017-10-02 18:38, Justin Palmer <justin@github.com> wrote:
> > > Hi Sai,
> > >
> > > At GitHub we had a similar issue with TriggerDagRunOperator.  In addition
> > > to scheduling tasks in the future, we wanted a stable execution date that
> > > was based on the triggering DAG.  We created a plugin that is basically a
> > > copy of TriggerDagRunOperator, except it passes in the `execution_date`
> > to
> > > the triggered DAG.   I've linked the plugin below.  Feel free to use it
> > > under the MIT license.
> > >
> > > https://gist.github.com/Caged/f356430518247d1bbc2439a153e3c79e
> > >
> > > -Justin
> > >
> > > On Mon, Oct 2, 2017 at 2:02 PM jspsai@gmail.com <jspsai@gmail.com>
> > wrote:
> > >
> > > > Hi experts,
> > > >
> > > > I am running the TriggerDagRunOperator example that comes with Airflow.
> > > >
> > > > example_trigger_controller_dag
> > > > example_trigger_target_dag
> > > >
> > > > I've created 2 new dags when are based on the above controller and
> > trigger
> > > > dags.
> > > >
> > > > when I run the example_trigger_controller_dag, i see that the
> > > > examle_trigger_target_dag is scheduled to run after a few hours later,
> > how
> > > > to schedule it to run immediately?
> > > > here is the logs where the Sub task that is scheduled to run at
> > 17:53:31
> > > > and the Base task has ran at 13:53:31.
> > > >
> > > > [2017-10-02 13:53:31,624] {base_task_runner.py:95} INFO - Subtask:
> > > > [2017-10-02 13:53:31,623] {dagrun_operator.py:74} INFO - Creating
> > DagRun
> > > > <DagRun Trigger_Target_Dag @ 2017-10-02 17:53:31:
> > > > trig__2017-10-02T13:53:31.248671, externally triggered: True>
> > > >
> > > > I might me doing something really stupid here.
> > > > please help.
> > > >
> > > > Thanks
> > > > -Sai.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > Justin,
> >
> > When i try to compile i am getting an import error for the plugin
> > This is how i am importing.
> > from airflow.operators.execute_dag_run_operator import
> > ExecuteDagRunOperator
> >
> > Below is the error when compiled.
> > ImportError: No module named execute_dag_run_operator
> > [2017-10-03 11:09:36,242] [22209] {models.py:266} ERROR - Failed to
> > import: /home/ec2-user/airflow/dags/Test_Controller_Job.py
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/site-packages/airflow/models.py", line 263, in
> > process_file
> >     m = imp.load_source(mod_name, filepath)
> >   File "/home/ec2-user/airflow/dags/Test_Controller_Job.py", line 43, in
> > <module>
> >     from airflow.operators.execute_dag_run_operator import
> > ExecuteDagRunOperator
> >
> > I am on Airflow 1.8,
> >
> > Do you have an example implementation for this?
> >
> > Thanks
> > Sai
>Thanks Justin!, i'll give it a try