airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bence Nagy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-29) Decrease the default `dagbag_import_timeout`
Date Fri, 06 May 2016 13:44:12 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274038#comment-15274038
] 

Bence Nagy commented on AIRFLOW-29:
-----------------------------------

[~sanand] See issue description:

{quote}
I feel like having a really low default setting would be a nice way to make sure users don't
do silly things like I did.
{quote}

I'm saying that if there's a DAG definition file that takes over one second to import, then
I'm 99% certain that it is implemented wrong (like, again, the [S3FileTransformOperator in
1.7.0|https://github.com/airbnb/airflow/blob/1.7.0/airflow/operators/s3_file_transform_operator.py#L60-L61]).
And if I'm doing something wrong, I'd prefer Airflow to let me know with an error message,
instead of silently getting really sluggish and starting to waste resources.

Basically this change is just to help newbies avoid a pitfall that I believe could be fairly
common — leaving expensive operations in the DAG definition file (such as database calls)
due to not realizing that it will be imported upon every scheduler run.

> Decrease the default `dagbag_import_timeout`
> --------------------------------------------
>
>                 Key: AIRFLOW-29
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-29
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Bence Nagy
>            Priority: Minor
>
> The default setting as of 1.7.0 is:
> {code}
> dagbag_import_timeout = 30
> {code}
> I don't think there's any reason for DAG imports to take over 1 second. I didn't always
know this though, and had a DAG earlier that made DB queries on each run making the scheduler
a lot slower than it should've been. I feel like having a really low default setting would
be a nice way to make sure users don't do silly things like I did, if coupled with helpful
error reporting.
> Original issue: https://github.com/airbnb/airflow/issues/1380



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message