airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andreas Merkel (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-387) Dropped postgres connections
Date Tue, 02 Aug 2016 15:22:20 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andreas Merkel updated AIRFLOW-387:
-----------------------------------
    Environment: 
Running aiflow with postgresql+psycopg2, Postgres 9.1.19 backend.


  was:
Running aiflow with {{postgresql+psycopg2}}, Postgres 9.1.19 backend.

{{airflow.cfg}}:

{code}
[core]
airflow_home = $PWD
dags_folder = $PWD/dags
base_log_folder = $PWD/airflow_logs
plugins_folder = $PWD/plugins
executor = LocalExecutor
sql_alchemy_conn = postgresql+psycopg2://postgres:postgres@10.128.5.58/airflow
parallelism = 32
dag_concurrency = 16
max_active_runs_per_dag = 16
load_examples = False
donot_pickle = False
fernet_key =  ; provided via environment
dags_are_paused_at_creation = False

[webserver]
expose_config = true
authenticate = False
filter_by_owner = False
workers = 4
worker_class = sync

[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5

[celery]
celeryd_concurrency = 1  ; Oddly enough, this is needed though we don't use celery.
{code}

Installed packages:

{code}
airflow==1.7.1.3
alembic==0.8.6
Babel==1.3
cee-syslog-handler==0.3.3
cffi==1.6.0
chartkick==0.4.2
croniter==0.3.12
cryptography==1.3.2
dill==0.2.5
docutils==0.12
enum34==1.1.6
Flask==0.10.1
Flask-Admin==1.4.0
Flask-Cache==0.13.1
Flask-Login==0.2.11
Flask-WTF==0.12
funcsigs==0.4
future==0.15.2
gunicorn==19.3.0
idna==2.1
ipaddress==1.0.16
itsdangerous==0.24
Jinja2==2.8
lockfile==0.12.2
Mako==1.0.4
Markdown==2.6.6
MarkupSafe==0.23
named-cee-logger==0.2.0
numpy==1.11.0
pandas==0.18.1
psycopg2==2.6.1
pyasn1==0.1.9
pycparser==2.14
Pygments==2.1.3
python-daemon==2.1.1
python-dateutil==2.5.3
python-editor==1.0.1
pytz==2016.4
requests==2.10.0
setproctitle==1.1.10
six==1.10.0
smoketest-dag==0.7
SQLAlchemy==1.0.13
thrift==0.9.3
Werkzeug==0.11.10
WTForms==2.1
{code}




> Dropped postgres connections
> ----------------------------
>
>                 Key: AIRFLOW-387
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-387
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: core
>    Affects Versions: Airflow 1.7.1.3
>         Environment: Running aiflow with postgresql+psycopg2, Postgres 9.1.19 backend.
>            Reporter: Andreas Merkel
>
> With the {{postgresql+psycopg2}} driver, we observe dropped connections in the database
logs on the postgres server:
> {code}
> 2016-08-02 14:49:26 UTC LOG:  could not receive data from client: Connection reset by
peer
> 2016-08-02 14:49:26 UTC LOG:  unexpected EOF on client connection
> {code}
> Three of these messages appear even if Airflow is started without any DAGs. If there
are DAGs, the message appears whenever the DAG is executed (even for simple DAGs like a Python
operator that does nothing).
> On the Airflow side, everything works fine; it's just the messages in the log indicating
that Airflow is somehow dropping the connections to Postgres without closing them properly.
> {{airflow.cfg}}:
> {code}
> [core]
> airflow_home = $PWD
> dags_folder = $PWD/dags
> base_log_folder = $PWD/airflow_logs
> plugins_folder = $PWD/plugins
> executor = LocalExecutor
> sql_alchemy_conn = postgresql+psycopg2://postgres:postgres@10.128.5.58/airflow
> parallelism = 32
> dag_concurrency = 16
> max_active_runs_per_dag = 16
> load_examples = False
> donot_pickle = False
> fernet_key =  ; provided via environment
> dags_are_paused_at_creation = False
> [webserver]
> expose_config = true
> authenticate = False
> filter_by_owner = False
> workers = 4
> worker_class = sync
> [scheduler]
> job_heartbeat_sec = 5
> scheduler_heartbeat_sec = 5
> [celery]
> celeryd_concurrency = 1  ; Oddly enough, this is needed though we don't use celery.
> {code}
> Installed packages:
> {code}
> airflow==1.7.1.3
> alembic==0.8.6
> Babel==1.3
> cee-syslog-handler==0.3.3
> cffi==1.6.0
> chartkick==0.4.2
> croniter==0.3.12
> cryptography==1.3.2
> dill==0.2.5
> docutils==0.12
> enum34==1.1.6
> Flask==0.10.1
> Flask-Admin==1.4.0
> Flask-Cache==0.13.1
> Flask-Login==0.2.11
> Flask-WTF==0.12
> funcsigs==0.4
> future==0.15.2
> gunicorn==19.3.0
> idna==2.1
> ipaddress==1.0.16
> itsdangerous==0.24
> Jinja2==2.8
> lockfile==0.12.2
> Mako==1.0.4
> Markdown==2.6.6
> MarkupSafe==0.23
> named-cee-logger==0.2.0
> numpy==1.11.0
> pandas==0.18.1
> psycopg2==2.6.1
> pyasn1==0.1.9
> pycparser==2.14
> Pygments==2.1.3
> python-daemon==2.1.1
> python-dateutil==2.5.3
> python-editor==1.0.1
> pytz==2016.4
> requests==2.10.0
> setproctitle==1.1.10
> six==1.10.0
> smoketest-dag==0.7
> SQLAlchemy==1.0.13
> thrift==0.9.3
> Werkzeug==0.11.10
> WTForms==2.1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message