airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Rolland <alexis.roll...@ubisoft.com>
Subject Close SqlSensor Connection
Date Tue, 16 Jan 2018 15:01:22 GMT
Hello everyone,

I'm reaching out to discuss / suggest a small improvement in the class SqlSensor:
https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html

We are currently using SqlSensors on top of Teradata in several DAGs. When the DAGs execute
we receive the following error message from Teradata engine: Error 8024 All virtual circuits
are currently in use.
This error message would typically appear when we reach the maximum number of simultaneous
connections to the database.

I am suspecting the SqlSensor task to create a lot of connections - basically every time it
(re)tries, and these connections would end up in idle state.
Does closing the connection at the end of the SqlSensor poke method sounds feasible?

I'd like to take this opportunity as well to thank you for the awesome work you've been doing
with Airflow.
Keep it up!

Best,

[far-cry-signature]


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message