airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Kegley <k...@b23.io>
Subject Re: Possible airflow-pool bug
Date Tue, 01 Nov 2016 16:10:16 GMT
Hey Gerard, 

For our use case we have 1 pool per dag and each dag has between 50 and 100 tasks.  We have
the pool size set between 3 and 5 in most cases and have seen up to 50 running tasks within
the pool

Best,
David

> On Nov 1, 2016, at 11:17 AM, Gerard Toonstra <gtoonstra@gmail.com> wrote:
> 
> David,
> 
> When you say "massive" oversubscribing, are you running a lot of dags in
> parallel that
> use your configured pools? Access to pools is not atomic at the moment.
> 
> Can you also quantify "massive" ?  Not that it matters, but to get a better
> idea.
> 
> Rgds,
> 
> Gerard
> 
> 
> On Tue, Nov 1, 2016 at 4:09 PM, siddharth anand <sanand@apache.org> wrote:
> 
>> Yes.. we have seen the over-subscription of pools. We do need a fix for it
>> --- I don't believe there is one. We need someone to own and fix it.. happy
>> to review a PR.
>> 
>> We use pools at Agari for all of our needs. We are okay with mild
>> oversubscription, so we do see numbers slightly higher, but our pipelines
>> work fine with that level. In your example, the oversubscription is much
>> higher.
>> -s
>> 
>> On Tue, Nov 1, 2016 at 6:38 AM, David Kegley <kegs@b23.io> wrote:
>> 
>>> I've been seeing some weird behavior when using Airflow's execution pool
>>> feature.  Pools have been massively over-filled leading to failed tasks.
>> I
>>> created a bug for this issue, but in the mean time, has anyone else
>>> experienced this behavior?
>>> 
>>> https://issues.apache.org/jira/browse/AIRFLOW-584
>>> 
>>> Best,
>>> David
>>> 
>> 


Mime
View raw message