Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 39502200D1A for ; Mon, 25 Sep 2017 07:44:23 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 37D281609E8; Mon, 25 Sep 2017 05:44:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 79DF41609E6 for ; Mon, 25 Sep 2017 07:44:22 +0200 (CEST) Received: (qmail 41726 invoked by uid 500); 25 Sep 2017 05:44:21 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 41714 invoked by uid 99); 25 Sep 2017 05:44:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Sep 2017 05:44:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A9A79DB501 for ; Mon, 25 Sep 2017 05:44:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.65 X-Spam-Level: X-Spam-Status: No, score=-0.65 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, HTML_OBFUSCATE_05_10=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id G9VZF8hivNZ4 for ; Mon, 25 Sep 2017 05:44:19 +0000 (UTC) Received: from mail-pf0-f182.google.com (mail-pf0-f182.google.com [209.85.192.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 37C735FBB0 for ; Mon, 25 Sep 2017 05:44:19 +0000 (UTC) Received: by mail-pf0-f182.google.com with SMTP id z84so3289709pfi.2 for ; Sun, 24 Sep 2017 22:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=jbPMzNFJXBegXO2X1eoBicctp5A+3GTYAAxMmoe/agE=; b=kD9SCOXpY9/E45fFHT7S7EDRS8//GLgh4btK4tLWMNsLjA7VbyvxyFPpfTa7LKn1n4 gG0cLsGKit+tKt9t2njhhQpwxNgpjBoI3f2H8fdW0FLz5TRmVwYjZZMQDkPegLsaj+9W QPscyIFbpOW4Ck8IacDt6ld6pwmri6Chs/7P23cdcSWO3tZwwsQFUTZT7aNW9nvjoFba SxnqVmSeiYvqrZYoXbZA1aWm6z/ATU7j7NkLl/SK/xNT2GW4SRdgEz6rDn2+qQgrnHb6 VyyWlMRl+12jfrX+lbChrSx8HYwkH5akTDshlDa130m/6qNLvWOsM+Gu8ROE3OMjJkRO W7tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=jbPMzNFJXBegXO2X1eoBicctp5A+3GTYAAxMmoe/agE=; b=WpjAzNknZ73Qkf29KdnCDR965schQs/IZXtTKHfQpOyBr+v6cUNZPujhtugiKjc+Wn pXVJ2H7WDgBzDoXVDvC05Zn3KrXUg/r9FSBpj4rejtfzEYLxvSWXJLN3q3VrHNrMLBOw rK45BxHdd+7HUhhUoq33QGY2QiQxp6kzqjwmk+cSrnxignYqHeqPx1V6SLwp0TMgYhnz GL/I9RljWKyOdm73Q4r1JQDZbEskrp95+BdqWOnGK6Zvp5FPBemH2vkgbue28gNaI4fm lRexzqwzgaAXaW1aICW7U/HOQSUwERHU3kyOHww8lsRWPxSbVGZ7iPQhE9xWtmkLLEWX KFLw== X-Gm-Message-State: AHPjjUjRNLE8FcIRDdYopv7AouZOz1GU17EqUiGm0D+Se9yzLhWqCvX3 6bvmUYFbk+2fdqWbI/pirvAhyduluC3ksHMeVovENWHZ X-Google-Smtp-Source: AOwi7QAv6h4tvm+Sr+eadGb5l5lUHnrtZ7uELTVGS7cXfTZ8W07ppBYzGNGnznhJ9hYMlZO6b5AP6QP7pvI0eSazYfU= X-Received: by 10.84.232.199 with SMTP id x7mr6621405plm.191.1506318257624; Sun, 24 Sep 2017 22:44:17 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.139.207 with HTTP; Sun, 24 Sep 2017 22:44:17 -0700 (PDT) From: Shoumitra Srivastava Date: Sun, 24 Sep 2017 22:44:17 -0700 Message-ID: Subject: Issues with postgres_operator To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="f4030436193caa1a0a0559fd0c9f" archived-at: Mon, 25 Sep 2017 05:44:23 -0000 --f4030436193caa1a0a0559fd0c9f Content-Type: text/plain; charset="UTF-8" Hey guys, I am currently using Airflow 1.8.2 to schedule some EMR tasks and then execute some long running queries on our Redshift cluster. For that purpose I am using the postgres_operator. The queries take about 30 minutes to run. However, once they are done, the connection never closes and the operator runs for an hour and a half more till its terminated at the 2 hour mark every time. The logs say that the server closed the connection unexpectedly. I've checked the logs on Redshift's end and it shows the queries have run and the connection has been closed. Somehow, that is never communicated back to Airflow. Any directions on what more I could check? To give some more info, my Airflow installation is an extension of the https://github.com/puckel/docker-airflow docker image, is run in an ECS cluster and has SQLite as backend since I am still testing Airflow out. I am also using Sequential Executor right now. I would appreciate any help in this matter. thanks, Shoumitra --f4030436193caa1a0a0559fd0c9f--