airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Close SqlSensor Connection
Date Tue, 16 Jan 2018 19:17:51 GMT
Good point Chris. I overlooked that.

B.

> On 16 Jan 2018, at 20:09, Chris Palmer <crispy16@gmail.com> wrote:
> 
> I'm not sure this is the right solution. I haven't explored all the code
> but it seems to me that the actual database connections are managed in the
> hooks. Different databases will want to handle this differently, and
> modifying SqlSensor seems too heavy handed to me.
> 
> 
> If you look at the get_records method of the generic DbApiHook (
> https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then
> you can see it does closer the connections it is making.
> 
> Is there specific hook for Teradata, and have you looked at how that is
> handling connections?
> 
> Chris
> 
> 
> On Jan 16, 2018 9:32 AM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:
> 
> Of course. PR welcome. It would be nice to know how to test for connection
> leakage.
> 
> Sent from my iPhone
> 
>> On 16 Jan 2018, at 16:01, Alexis Rolland <alexis.rolland@ubisoft.com>
> wrote:
>> 
>> Hello everyone,
>> 
>> I’m reaching out to discuss / suggest a small improvement in the class
> SqlSensor:
>> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>> 
>> We are currently using SqlSensors on top of Teradata in several DAGs.
> When the DAGs execute we receive the following error message from Teradata
> engine: Error 8024 All virtual circuits are currently in use.
>> This error message would typically appear when we reach the maximum
> number of simultaneous connections to the database.
>> 
>> I am suspecting the SqlSensor task to create a lot of connections -
> basically every time it (re)tries, and these connections would end up in
> idle state.
>> Does closing the connection at the end of the SqlSensor poke method
> sounds feasible?
>> 
>> I’d like to take this opportunity as well to thank you for the awesome
> work you’ve been doing with Airflow.
>> Keep it up!
>> 
>> Best,
>> 
>> 
>> 


Mime
View raw message