airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Greene <>
Subject Re: Custom scheduler support in Airflow
Date Thu, 31 Jan 2019 20:14:15 GMT
This is actually the majority of our airflow work.

We use this pattern:
Sensor pings api (fairly quickly, in a dag that’s constrained to only run one instance,
every minute)

If the sensor gets a valid response, the next task is a custom operator that extends Trigger,
builds up the DagRun context, and triggers the actual processing dag.

This has worked very well, and doesn’t require a custom scheduler.  Additionally, the “trigger
dags” are fairly light on resources- 1 active sensor each.

Sent from a device with less than stellar autocorrect

> On Jan 31, 2019, at 10:55 AM, abhishek sharma <> wrote:
> Hi Ben,
> Just copying my comment form ticket.
> I think current airflow scheduler schedule DAGs only on time-basis (based
> on cron schedule string). ***Is it correct understanding?*
> How to approach a scenario where I want to trigger a DAG based on some
> event which is not so predictable/regular on time basis?
>   - One is to use sensor and DAG flow will start by first running a sensor
>   task for checking an event and when an event happened actual processing get
>   a start.
>   - Second, is to have a DAG with None schedule and it gets triggered by
>   some other application or utility which checks for an event occurrence .
> If your most of the flow(DAGs) are suppose to run on this logic, then
> sensor doesn't make sense and hence left with the second approach only,
> which is nothing but a custom scheduling of DAGs.  ****Is it agreeable use
> case for the custom scheduler?*
> Thanks, {{Abhishek}}
>> On Thu, Jan 31, 2019 at 11:53 AM Ben Tallman <> wrote:
>> Can you explain a bit more what you are thinking for a custom scheduler?
>> It's been awhile, but we added support for cron schedules without backfill
>> awhile back, so I'm wondering what you are thinking of adding with this?
>> Thanks,
>> Ben
>> --
>> Ben Tallman - 503.680.5709
>> On Thu, Jan 31, 2019 at 8:29 AM abhishek sharma <
>> wrote:
>>> Hi All,
>>> Created a ticket( for
>>> supporting custom scheduler in Airflow.
>>> The idea is to have a scheduler base class which can be extended for
>>> writing a custom scheduler. The logic of custom scheduling is user
>>> specific, and at the DAGs task level we can mention the scheduler type,
>> and
>>> that scheduler will be used for starting a task. [Naive Idea]
>>> Can we please discuss whether we need such functionality in Airflow or
>> not
>>> and If yes then we will proceed with the design and implementation.
>>> Thanks
>>> Abhishek Sharma

View raw message