airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-2442) Airflow run command leaves database connections open, which can hit the database limit
Date Wed, 09 May 2018 01:36:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alejandro Fernandez updated AIRFLOW-2442:
-----------------------------------------
    Description: 
*Summary*

The "airflow run" command creates a connection to the database and leaves it open (until killed
by SQLALchemy later). The number of these connections can skyrocket whenever hundreds/thousands
of tasks are launched simultaneously, and potentially hit the database connection limit.

The problem is that in cli.py, the run() method first calls 
{code:java}
settings.configure_orm(disable_connection_pool=True){code}
correctly
 to use a NullPool, but then parses any custom configs and again calls
{code:java}
settings.configure_orm(){code}
, thereby overriding the desired behavior by instead using a QueuePool.
 The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE and SQL_ALCHEMY_POOL_RECYCLE.
This means that while the task is running and the executor is sending heartbeats, the sleeping
connection is idle until it is killed by SQLAlchemy.

This fixes a bug introduced by [https://github.com/apache/incubator-airflow/pull/1934] in
[https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]  

which is present in branches 1-8-stable, 1-9-stable, and 1-10-test

  was:
*Summary*

The "airflow run" command creates a connection to the database and leaves it open (until killed
by SQLALchemy later). The number of these connections can skyrocket whenever hundreds/thousands
of tasks are launched simultaneously, and potentially hit the database connection limit.

The problem is that in cli.py, the run() method first calls 
{code}settings.configure_orm(disable_connection_pool=True)\{code} correctly
to use a NullPool, but then parses any custom configs and again calls \{code}settings.configure_orm()\{code},
thereby overriding the desired behavior with a QueuePool.
The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE (5 connections) and SQL_ALCHEMY_POOL_RECYCLE
(1 hour). This means that while the task is running and the executor is sending heartbeats,
the sleeping connection is idle until it is killed by SQLAlchemy.


This fixes a bug introduced by [https://github.com/apache/incubator-airflow/pull/1934] in
[https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]  

which is present in branches 1-8-stable, 1-9-stable, and 1-10-test


> Airflow run command leaves database connections open, which can hit the database limit
> --------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2442
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2442
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: cli
>    Affects Versions: Airflow 1.8, 1.8.0
>            Reporter: Alejandro Fernandez
>            Assignee: Alejandro Fernandez
>            Priority: Major
>             Fix For: Airflow 2.0
>
>         Attachments: connection_duration_1_hour.png, db_connections.png, monthly_db_connections.png,
running_tasks.png
>
>
> *Summary*
> The "airflow run" command creates a connection to the database and leaves it open (until
killed by SQLALchemy later). The number of these connections can skyrocket whenever hundreds/thousands
of tasks are launched simultaneously, and potentially hit the database connection limit.
> The problem is that in cli.py, the run() method first calls 
> {code:java}
> settings.configure_orm(disable_connection_pool=True){code}
> correctly
>  to use a NullPool, but then parses any custom configs and again calls
> {code:java}
> settings.configure_orm(){code}
> , thereby overriding the desired behavior by instead using a QueuePool.
>  The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE and SQL_ALCHEMY_POOL_RECYCLE.
This means that while the task is running and the executor is sending heartbeats, the sleeping
connection is idle until it is killed by SQLAlchemy.
> This fixes a bug introduced by [https://github.com/apache/incubator-airflow/pull/1934] in
[https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]  
> which is present in branches 1-8-stable, 1-9-stable, and 1-10-test



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message