airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shoumitra Srivastava <shoumitra...@gmail.com>
Subject Re: Issues with postgres_operator
Date Mon, 25 Sep 2017 18:09:18 GMT
Hi guys,

Thank you for your quick replies. Seems like Arthur's pull request
addresses exactly what I am facing. Hopefully, it gets merged and I can
move ahead.

thanks,
Shoumitra

On Mon, Sep 25, 2017 at 8:25 AM, Arthur Purvis <apurvis@lumoslabs.com>
wrote:

> fixed here: https://github.com/apache/incubator-airflow/pull/2513
>
> On Mon, Sep 25, 2017 at 10:28 AM, Driesprong, Fokko <fokko@driesprong.frl>
> wrote:
>
>> Hi Shoumitra,
>>
>> So if I understand correctly, the query finishes but the connection does
>> not close. What is the result of the query? Maybe you can also share some
>> code of how you use the PostgresOperator the communicate with Redshift.
>>
>> Regarding your config. The sequential executor is only recommended when
>> debugging, since it runs in process and only allows you to execute one
>> task
>> at the time. I would recommend upgrading to the LocalExecutor and use an
>> AWS managed Postgres for the database.
>>
>> Let me know.
>>
>> Cheers, Fokko
>>
>> 2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <shoumitra362@gmail.com>:
>>
>> > Hey guys,
>> >
>> > I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
>> > execute some long running queries on our Redshift cluster. For that
>> purpose
>> > I am using the postgres_operator. The queries take about 30 minutes to
>> run.
>> > However, once they are done, the connection never closes and the
>> operator
>> > runs for an hour and a half more till its terminated at the 2 hour mark
>> > every time. The logs say that the server closed the connection
>> > unexpectedly. I've checked the logs on Redshift's end and it shows the
>> > queries have run and the connection has been closed. Somehow, that is
>> never
>> > communicated back to Airflow. Any directions on what more I could
>> check? To
>> > give some more info, my Airflow installation is an extension of the
>> > https://github.com/puckel/docker-airflow docker image, is run in an ECS
>> > cluster and has SQLite as backend since I am still testing Airflow out.
>> I
>> > am also using Sequential Executor right now. I would appreciate any
>> help in
>> > this matter.
>> >
>> > thanks,
>> > Shoumitra
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message