From dev-return-9200-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Fri Aug 16 15:53:11 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 53FAA18057A for ; Fri, 16 Aug 2019 17:53:11 +0200 (CEST) Received: (qmail 38608 invoked by uid 500); 16 Aug 2019 15:53:10 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 38596 invoked by uid 99); 16 Aug 2019 15:53:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Aug 2019 15:53:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BE8CAC1CE2 for ; Fri, 16 Aug 2019 15:53:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.501 X-Spam-Level: X-Spam-Status: No, score=-1.501 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id jsuCzYYfBRe8 for ; Fri, 16 Aug 2019 15:53:08 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=198.240.130.133; helo=mail-gw29.credit-suisse.com; envelope-from=damian.shaw.2@credit-suisse.com; receiver= Received: from mail-gw29.credit-suisse.com (mail-gw29.credit-suisse.com [198.240.130.133]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id C9806BC7A9 for ; Fri, 16 Aug 2019 15:53:07 +0000 (UTC) Received: by mail-gw29.credit-suisse.com; Fri, 16 Aug 2019 15:52:59 GMT From: "Shaw, Damian P. " To: "dev@airflow.apache.org" Subject: RE: Outage report Thread-Topic: Outage report Thread-Index: AQHVVEctCNCOvUDOrkKSGfkwCu+sIab96FUQ Date: Fri, 16 Aug 2019 15:52:53 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-sentfromoutlook: SentFromOutlook Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 Thanks, this is really useful to know! I often write my own Operators/Sensors/Hooks and was just looking at doing the same with the SFTPSensor and Operator. I've never formalized it but my current pattern is the follow: Hooks, Set self._conn to None on __init__, and have a property "self.conn" that checks if "self._conn" is None, *if None create a new connection set it to self._conn and return it * if not None run a check to see if the connection is still alive, if is alive return self._conn, otherwise create a new connection Sensor/Operators, On __init__ set self.conn_id to the conn_id string, and set "self._{conn_type}_hook" to None and have a property "self.{conn_type}_hook" In property check if "self._{conn_type}_hook" is None and if so create a new Hook, if not None then return "self._{conn_type}_hook" I would be really appreciative on any best practices here others could share. -----Original Message----- From: James Meickle [mailto:jmeickle@quantopian.com.INVALID] Sent: Friday, August 16, 2019 11:27 AM To: dev@airflow.apache.org Subject: Outage report We had an outage last night that was rather complex and difficult to debug. Rather than just writing up the bug, I included what we did for various debug steps. Hope some folks who are also cluster maintainers may find it interesting! https://issues.apache.org/jira/browse/AIRFLOW-5238 =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================