airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-2371) SubdagOperator ignores pooling when not using SequentialExecutor
Date Tue, 24 Apr 2018 22:09:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiao Zhu updated AIRFLOW-2371:
------------------------------
    Description: 
Basically we need a long term solution for https://issues.apache.org/jira/browse/AIRFLOW-74
. Using SequentialExecutor is a workaround, however it slows down execution of subdags.
Looking for a better solution.

I also see that when running subdags, db connection pools are not shared either (since they
are separate processes), which causes db disconnection issues like https://issues.apache.org/jira/browse/AIRFLOW-2340
on our db because there can be too many connections. I hope it can be fixed too when subdag
operator works with pooling..

  was:
Basically we need a long term solution for https://issues.apache.org/jira/browse/AIRFLOW-74
. Using SequentialExecutor is a workaround, however it slows down execution of subdags.
Looking for a better solution.

I also see that when running subdags, db connection pools are not shared either (since they
are separate processes), which causes db disconnection issues like https://issues.apache.org/jira/browse/AIRFLOW-2340
on our db because there can be too many connections. I hope it can be fixed when subdag operator
can work with pooling!



> SubdagOperator ignores pooling when not using SequentialExecutor
> ----------------------------------------------------------------
>
>                 Key: AIRFLOW-2371
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2371
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: subdag
>    Affects Versions: Airflow 1.9.0
>         Environment: Airflow 1.9.0
>            Reporter: Xiao Zhu
>            Priority: Major
>
> Basically we need a long term solution for https://issues.apache.org/jira/browse/AIRFLOW-74
. Using SequentialExecutor is a workaround, however it slows down execution of subdags.
> Looking for a better solution.
> I also see that when running subdags, db connection pools are not shared either (since
they are separate processes), which causes db disconnection issues like https://issues.apache.org/jira/browse/AIRFLOW-2340
on our db because there can be too many connections. I hope it can be fixed too when subdag
operator works with pooling..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message