airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shoumitra Srivastava <shoumitra...@gmail.com>
Subject Issues with postgres_operator
Date Mon, 25 Sep 2017 05:44:17 GMT
Hey guys,

I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
execute some long running queries on our Redshift cluster. For that purpose
I am using the postgres_operator. The queries take about 30 minutes to run.
However, once they are done, the connection never closes and the operator
runs for an hour and a half more till its terminated at the 2 hour mark
every time. The logs say that the server closed the connection
unexpectedly. I've checked the logs on Redshift's end and it shows the
queries have run and the connection has been closed. Somehow, that is never
communicated back to Airflow. Any directions on what more I could check? To
give some more info, my Airflow installation is an extension of the
https://github.com/puckel/docker-airflow docker image, is run in an ECS
cluster and has SQLite as backend since I am still testing Airflow out. I
am also using Sequential Executor right now. I would appreciate any help in
this matter.

thanks,
Shoumitra

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message