airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1559) MySQL warnings about aborted connections, missing engine disposal
Date Sat, 02 Dec 2017 15:24:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275602#comment-16275602
] 

ASF subversion and git services commented on AIRFLOW-1559:
----------------------------------------------------------

Commit 5a303ebbc572cee7c9c30be84ebf625357360d4b in incubator-airflow's branch refs/heads/master
from [~StephanErb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5a303eb ]

[AIRFLOW-1559] Close file handles in subprocesses

All file descriptors except 0, 1 and 2 will be closed before the
child process is executed. This is the default on Python 3.2 and
above. This patch ensures consistent behaviour for older Python
versions.

Resources will be released once the main thread disposes
them, independent of the longevity of its subprocesses.

Background information:

* https://www.python.org/dev/peps/pep-0446/
* https://bugs.python.org/issue7213


> MySQL warnings about aborted connections, missing engine disposal
> -----------------------------------------------------------------
>
>                 Key: AIRFLOW-1559
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1559
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db
>            Reporter: Daniel Huang
>            Assignee: Daniel Huang
>            Priority: Minor
>             Fix For: 1.9.1
>
>
> We're seeing a flood of warnings about aborted connections in our MySQL logs. 
> {code}
> Aborted connection 56720 to db: 'airflow' user: 'foo' host: 'x.x.x.x' (Got an error reading
communication packets)
> {code}
> It appears this is because we're not performing [engine disposal|http://docs.sqlalchemy.org/en/latest/core/connections.html#engine-disposal].
The most common source of this warning is from the scheduler, when it kicks off new processes
to process the DAG files. Calling dispose in https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L403
greatly reduced these messages. However, the worker is still causing some of these, I assume
from when we spin up processes to run tasks. We do call dispose in https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L1394-L1396,
but I think it's a bit early. Not sure if there's a place we can put this cleanup to ensure
it's done everywhere.
> Quick script to reproduce this warning message:
> {code}
> from airflow import settings
> from airflow.models import Connection
> session = settings.Session()
> session.query(Connection).count()
> session.close()
> # not calling settings.engine.dispose()
> {code}
> Reproduced with Airflow 1.8.1, MySQL 5.7, and SQLAlchemy 1.1.13. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message