airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Close SqlSensor Connection
Date Tue, 16 Jan 2018 17:32:53 GMT
Of course. PR welcome. It would be nice to know how to test for connection leakage. 

Sent from my iPhone

> On 16 Jan 2018, at 16:01, Alexis Rolland <alexis.rolland@ubisoft.com> wrote:
> 
> Hello everyone,
>  
> I’m reaching out to discuss / suggest a small improvement in the class SqlSensor:
> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>  
> We are currently using SqlSensors on top of Teradata in several DAGs. When the DAGs execute
we receive the following error message from Teradata engine: Error 8024 All virtual circuits
are currently in use.
> This error message would typically appear when we reach the maximum number of simultaneous
connections to the database.
>  
> I am suspecting the SqlSensor task to create a lot of connections - basically every time
it (re)tries, and these connections would end up in idle state.
> Does closing the connection at the end of the SqlSensor poke method sounds feasible?
>  
> I’d like to take this opportunity as well to thank you for the awesome work you’ve
been doing with Airflow.
> Keep it up!
>  
> Best,
>  
> 
>  

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message