airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Coder <jcode...@gmail.com>
Subject Re: Schedule interval options
Date Fri, 10 May 2019 08:25:06 GMT
I think I originally dismissed that idea because I was thinking in terms in prev_ds. If your
do 2-6 then on 2, you’re prev_ds in the TI context is 6, but if you just use yesterday_ds
that is just crazy enough to work. 
I can’t believe it took me this long to realize that. 

James Coder

> On May 10, 2019, at 4:05 AM, Bas Harenslak <basharenslak@godatadriven.com> wrote:
> 
> Yea that’s why I also suggested 2-6.
> 
> If that does not work for you, could you explain the use case?
> 
> Bas
> 
>> On 9 May 2019, at 22:00, James Coder <jcoder01@gmail.com> wrote:
>> 
>> Yes, but doing 1-5 would not run 5 until 1.
>> 
>> James Coder
>> 
>>> On May 9, 2019, at 2:40 PM, Bas Harenslak <basharenslak@godatadriven.com>
wrote:
>>> 
>>> Perhaps needless to say, but you can do this with a cron expression, e.g. “0
0 * * 1-5” to schedule on weekdays only. Or “0 0 * * 2-6” to ensure you start running
from Tuesday 00:00 to Saturday 00:00 so you also process Fridays data.
>>> 
>>> Does that help?
>>> 
>>> Bas
>>> 
>>> On 9 May 2019, at 18:53, James Coder <jcoder01@gmail.com<mailto:jcoder01@gmail.com>>
wrote:
>>> 
>>> Hi All,
>>> I feel like this has probably been discussed more times than necessary, but
>>> I wanted to get the community opinion on running dags for business days. In
>>> my case I want to run M-F but I don't want to wait for Monday to run
>>> Fridays data. As far as I can tell the only way to do this right now is to
>>> schedule it to run everyday and short circuit on Saturday and Sunday.
>>> I would like to explore options other than adding an additional operator to
>>> all dags.
>>> I have seen it suggested to add a "schedule type" argument to dag that
>>> would allow for running at "the right side or left side" of the interval.
>>> Other options I can think of to solve for this would be specify add an
>>> additional dag argument that allows you to explicitly set the interval
>>> length rather than relying on cronitor to provide the following schedule.
>>> The last option I can think of would be to add support for passing a
>>> function as schedule_interval (a function that returns a time delta).
>>> 
>>> To summarize possible changes:
>>> 1. add "schedule_type" kwarg (start or end)
>>> 2. add "interval_length" kwarg (time delta that is used in conjunction with
>>> cronitor)
>>> 3. support functions as schedule_interval.
>>> 
>>> Thoughts?
>>> 
>>> Thanks,
>>> James
>>> 
> 

Mime
View raw message