airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Rolland <alexis.roll...@ubisoft.com>
Subject Re: Close SqlSensor Connection
Date Wed, 17 Jan 2018 00:48:10 GMT
Ok indeed I can see the connection is getting closed in get_records!
Thanks

Alexis Rolland
+86 138 1602 1449

> Le 17 janv. 2018 à 03:18, Bolke de Bruin <bdbruin@gmail.com> a écrit :
> 
> Good point Chris. I overlooked that.
> 
> B.
> 
>> On 16 Jan 2018, at 20:09, Chris Palmer <crispy16@gmail.com> wrote:
>> 
>> I'm not sure this is the right solution. I haven't explored all the code
>> but it seems to me that the actual database connections are managed in the
>> hooks. Different databases will want to handle this differently, and
>> modifying SqlSensor seems too heavy handed to me.
>> 
>> 
>> If you look at the get_records method of the generic DbApiHook (
>> https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then
>> you can see it does closer the connections it is making.
>> 
>> Is there specific hook for Teradata, and have you looked at how that is
>> handling connections?
>> 
>> Chris
>> 
>> 
>> On Jan 16, 2018 9:32 AM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:
>> 
>> Of course. PR welcome. It would be nice to know how to test for connection
>> leakage.
>> 
>> Sent from my iPhone
>> 
>>> On 16 Jan 2018, at 16:01, Alexis Rolland <alexis.rolland@ubisoft.com>
>> wrote:
>>> 
>>> Hello everyone,
>>> 
>>> I’m reaching out to discuss / suggest a small improvement in the class
>> SqlSensor:
>>> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>>> 
>>> We are currently using SqlSensors on top of Teradata in several DAGs.
>> When the DAGs execute we receive the following error message from Teradata
>> engine: Error 8024 All virtual circuits are currently in use.
>>> This error message would typically appear when we reach the maximum
>> number of simultaneous connections to the database.
>>> 
>>> I am suspecting the SqlSensor task to create a lot of connections -
>> basically every time it (re)tries, and these connections would end up in
>> idle state.
>>> Does closing the connection at the end of the SqlSensor poke method
>> sounds feasible?
>>> 
>>> I’d like to take this opportunity as well to thank you for the awesome
>> work you’ve been doing with Airflow.
>>> Keep it up!
>>> 
>>> Best,
>>> 
>>> 
>>> 
> 
Mime
View raw message