airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Seelmann <m...@stefan-seelmann.de>
Subject Re: ExternalTaskSensor alternatives
Date Wed, 22 Aug 2018 19:10:11 GMT
On 08/22/2018 06:56 PM, Tao Feng wrote:
> FYI, there is an existing pr and proposal for improving sensor efficiency(
> https://issues.apache.org/jira/browse/AIRFLOW-2747 and
> https://github.com/apache/incubator-airflow/pull/3596/files) by the
> community.

And I hope I'll find some time next week to continue working on it. More
feedback and help is welcomed :)

> For your idea, I am not sure if it is a good idea to add this dag dependent
> functionality to the existing scheduler given its logic is already quite
> complicated.
> 
> On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard <
> emmanuel.brard@getyourguide.com> wrote:
> 
>> Hi everyone,
>>
>> I've recently looked at the implementation of the ExternalTaskSensor sensor
>> and I was wondering if it would be a good idea to actually implement this
>> check (these checks) at the scheduler level. Basically the
>> ExternalTaskSensor runs a query against the backend database at regular
>> intervals to know if the rule(s) have passed. This is also what the
>> scheduler does to some extent checking against all task instances deps. So
>> I though of something like adding the same (or similar) set of parameters
>> of the ExternalTaskSensor directly to the operators directly and have the
>> scheduler check for this.
>>
>> The idea around this is that the ExternalTaskSensors use resources to not
>> do much actually.
>>
>> I have read somewhere that having the scheduler executing these kind of
>> task could be a potential feature, it would already be a improvement but I
>> suppose we would need the scheduler to "thread out" to do so, and this will
>> just move the resources waste from the "executors" to the scheduler.
>>
>> What do you think ?
>>
>> Cheers,
>> Emmanuel
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>> GetYourGuide AG
>>
>> Stampfenbachstrasse 48
>>
>> 8006 Z├╝rich
>>
>> Switzerland
>>
>>
>>
>>  <https://www.facebook.com/GetYourGuide>
>> <https://twitter.com/GetYourGuide>
>> <https://www.instagram.com/getyourguide/>
>> <https://www.linkedin.com/company/getyourguide-ag>
>> <http://www.getyourguide.com>
>>
>>
>>
>>
>>
>>
>>
>>
> 


Mime
View raw message