airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-191) Database connection leak on Postgresql backend
Date Thu, 02 Jun 2016 19:27:59 GMT


Chris Riccomini commented on AIRFLOW-191:

Maybe they could all send their heartbeats to a service that runs on the manager node, and
the service can manage a modestly sized connection pool?"

Long term, the REST service should handle the heartbeats, so everything can heartbeat to the
service, and it can talk to the DB from one (or a few) places.

> Database connection leak on Postgresql backend
> ----------------------------------------------
>                 Key: AIRFLOW-191
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: executor
>    Affects Versions: Airflow
>            Reporter: Sergei Iakhnin
>         Attachments: Sid_anands_airflow_idle_in_transaction.png
> I raised this issue on github several months ago and there was even a PR but it never
maid it into mainline. Basically, workers tend to hang onto DB connections in Postgres for
recording heartbeat.
> I'm running a cluster with 115 workers, each with 8 slots. My Postgres DB is configured
to allow 1000 simultaneous connections. I should effectively be able to run 920 tasks at the
same time, but am actually limited to only about 450-480 because of idle transactions from
workers hanging on to DB connections.
> If I run the following query
> select count(*),state, client_hostname from pg_stat_activity group by state, client_hostname
> These are the results:
> count state client_hostname
> 1	active	(null)
> 1	idle	localhost
> 451	idle in transaction	(null)
> 446	idle	(null)
> 1	active	localhost
> The idle connections are all trying to run COMMIT
> The "idle in transaction" connections are all trying to run 
> SELECT AS job_id, job.dag_id AS job_dag_id, job.state AS job_state, job.job_type
AS job_job_type, job.start_date AS job_start_date, job.end_date AS job_end_date, job.latest_heartbeat
AS job_latest_heartbeat, job.executor_class AS job_executor_class, job.hostname AS job_hostname,
job.unixname AS job_unixname 
> FROM job 
> WHERE = 213823 
>  LIMIT 1
> with differing job.ids of course.

This message was sent by Atlassian JIRA

View raw message