airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-29) Decrease the default `dagbag_import_timeout`
Date Tue, 03 May 2016 15:18:13 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268863#comment-15268863
] 

Chris Riccomini commented on AIRFLOW-29:
----------------------------------------

> the most complex one is defining 5 DAGs with 4 tasks each

We have a .py that generates > 200 DAGs with 4 2-4 operators each.

> Decrease the default `dagbag_import_timeout`
> --------------------------------------------
>
>                 Key: AIRFLOW-29
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-29
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Bence Nagy
>            Priority: Minor
>
> The default setting as of 1.7.0 is:
> {code}
> dagbag_import_timeout = 30
> {code}
> I don't think there's any reason for DAG imports to take over 1 second. I didn't always
know this though, and had a DAG earlier that made DB queries on each run making the scheduler
a lot slower than it should've been. I feel like having a really low default setting would
be a nice way to make sure users don't do silly things like I did, if coupled with helpful
error reporting.
> Original issue: https://github.com/airbnb/airflow/issues/1380



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message