airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bas Harenslak <basharens...@godatadriven.com>
Subject Re: Outage report
Date Sat, 17 Aug 2019 09:16:50 GMT
Nice work! Always love reading these sort of “bug reports from hell” and the work required
to find the cause.

Also strongly agree we should standardize hooks in some way.

Cheers,
Bas

> On 16 Aug 2019, at 17:52, Shaw, Damian P. <damian.shaw.2@credit-suisse.com> wrote:
> 
> Thanks, this is really useful to know!  I often write my own Operators/Sensors/Hooks
and was just looking at doing the same with the SFTPSensor and Operator.
> 
> I've never formalized it but my current pattern is the follow:
> 
> Hooks,
> Set self._conn to None on __init__, and have a property "self.conn" that checks if "self._conn"
is None,
> *if None create a new connection set it to self._conn and return it
> * if not None run a check to see if the connection is still alive, if is alive return
self._conn, otherwise create a new connection 
> 
> Sensor/Operators,
> On  __init__ set self.conn_id to the conn_id string, and set "self._{conn_type}_hook"
to None and have a property "self.{conn_type}_hook"
> In property check if "self._{conn_type}_hook" is None and if so create a new Hook, if
not None then return "self._{conn_type}_hook"
> 
> I would be really appreciative on any  best practices here others could share. 
> 
> 
> -----Original Message-----
> From: James Meickle [mailto:jmeickle@quantopian.com.INVALID] 
> Sent: Friday, August 16, 2019 11:27 AM
> To: dev@airflow.apache.org
> Subject: Outage report
> 
> We had an outage last night that was rather complex and difficult to debug.
> Rather than just writing up the bug, I included what we did for various
> debug steps. Hope some folks who are also cluster maintainers may find it
> interesting!
> 
> https://issues.apache.org/jira/browse/AIRFLOW-5238
> 
> 
> 
> =============================================================================== 
> Please access the attached hyperlink for an important electronic communications disclaimer:

> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
> =============================================================================== 

Mime
View raw message