airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaw, Damian P. " <damian.sha...@credit-suisse.com>
Subject RE: Outage report
Date Fri, 16 Aug 2019 15:52:53 GMT
Thanks, this is really useful to know!  I often write my own Operators/Sensors/Hooks and was
just looking at doing the same with the SFTPSensor and Operator.

I've never formalized it but my current pattern is the follow:

Hooks,
Set self._conn to None on __init__, and have a property "self.conn" that checks if "self._conn"
is None,
 *if None create a new connection set it to self._conn and return it
 * if not None run a check to see if the connection is still alive, if is alive return self._conn,
otherwise create a new connection 

Sensor/Operators,
On  __init__ set self.conn_id to the conn_id string, and set "self._{conn_type}_hook" to None
and have a property "self.{conn_type}_hook"
In property check if "self._{conn_type}_hook" is None and if so create a new Hook, if not
None then return "self._{conn_type}_hook"

I would be really appreciative on any  best practices here others could share. 


-----Original Message-----
From: James Meickle [mailto:jmeickle@quantopian.com.INVALID] 
Sent: Friday, August 16, 2019 11:27 AM
To: dev@airflow.apache.org
Subject: Outage report

We had an outage last night that was rather complex and difficult to debug.
Rather than just writing up the bug, I included what we did for various
debug steps. Hope some folks who are also cluster maintainers may find it
interesting!

https://issues.apache.org/jira/browse/AIRFLOW-5238



=============================================================================== 
Please access the attached hyperlink for an important electronic communications disclaimer:

http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
=============================================================================== 

Mime
View raw message