airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Galak (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-2010) Make HttpHook inner connection pool configurable
Date Wed, 17 Jan 2018 21:07:00 GMT
Galak created AIRFLOW-2010:

             Summary: Make HttpHook inner connection pool configurable
                 Key: AIRFLOW-2010
             Project: Apache Airflow
          Issue Type: Improvement
          Components: hooks
    Affects Versions: 1.8.0
            Reporter: Galak

HttpHook is using request module to perform http/https calls. but it is hidden inside implementation.
Therefore, it is not possible to choose any value for _pool_connections_ orĀ  _pool_maxsize_
parameters, defaulting to 10. (see [request module documentation|])

_{{requests.adapters.HTTPAdapter}}_ parameters could probably be passed through Connection
extra parameters ?

As a consequence, calling a REST API concurrently (using [ThreadPoolExecutor|])
is limited to 10 workers maximim. Each additional worker is stopped with the following warning:
{quote}{{{} WARNING - Connection pool is full, discarding connection:}}{quote}
See [this question on stackoverflow|]
about Http connexion pools configuration

This message was sent by Atlassian JIRA

View raw message