airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Edmiston <tedmis...@gmail.com>
Subject Re: Regarding airflow scheduler starting PID periodically.
Date Thu, 10 May 2018 07:45:50 GMT
If it is different than the TriggerDagRunOperator idea that Guillermo
mentioned, could you elaborate on your specific use case?

On Thu, May 10, 2018 at 2:40 AM Guillermo Rodríguez Cano <wschutz@gmail.com>
wrote:

> Hello,
>
> I am not sure if I understood... you want to trigger DAGs based on the PIDs
> of the tasks?
>
> As you say (I am not an expert on OSes here) you don't control this.
>
> I think that the TriggerDagRunOperator (https://airflow.apache.org/co
> de.html#airflow.operators.TriggerDagRunOperator
> <https://airflow.apache.org/code.html#airflow.operators.TriggerDagRunOperator>)
> would work for you. Right?
>
> /Guillermo
>
>
> On Tue, May 8, 2018 at 5:06 PM nitish sharma <nitishcse412@gmail.com>
> wrote:
>
> > Hi Team,
> >
> > I have a situation where i want to create dag with unique dag id and run
> > that dag once as soon as it is created.
> >
> > For eg :  1.py , 2.py are two different dags. 1.py (has 3 tasks) and
> > 2.py(has 2 tasks).
> >
> > Once these are run i would have next set of dags with me for eg 3.py,
> 4.py,
> > 5.py and so on.
> >
> > I see a challenger here as scheduler keeps generating process ID to
> > generate tasks
> >
> >
>
-- 
*Taylor Edmiston*
Blog <http://blog.tedmiston.com> | Stack Overflow CV
<https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message