airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaw, Damian P. " <damian.sha...@credit-suisse.com>
Subject RE: How to manage Airflow SIGTERM excetion catches?
Date Thu, 03 Oct 2019 15:08:37 GMT
Hi Alex,

Maybe I'm stating the obvious but if any executors catches Airflow exceptions to do cleanup
before exiting they can still do that with SystemExit. E.g.:

class AirflowSystemExit(SystemExit):
    pass

try:
    raise AirflowSystemExit
except AirflowSystemExit:
    print('cleanup for airflow throwing system exit')
finally:
    print('cleanup for any type of exit')

In fact if people use either:

try:
    ...
except:
    ...

Or

try:
    ...
finally:
    ...

Their code will still run on this type of exit, but in the case of 1) this can at least be
put down to poor python code and case 2) any code in a finally statement shouldn't be written
so it can cause harm on exiting.

Damian


-----Original Message-----
From: Alex Guziel [mailto:alex.guziel@airbnb.com.INVALID] 
Sent: Wednesday, October 02, 2019 4:03 PM
To: dev@airflow.apache.org
Subject: Re: How to manage Airflow SIGTERM excetion catches?

Task_copy.on_kill() should probably be killing the underlying process, but
I think it's fuzzy where the exception gets thrown. I think the intention
is for the exception to get caught in that same block, so the cleanup can
happen, but this is not the case since it is thrown in the main thread. I
think the behavior depends a little bit on which executor you use. We
probably need to think about the right behavior here that works for a
variety of executors. SystemExit works for things that don't use
subprocess, but I'm not sure what the behavior for cleanup is when running
subprocess.



=============================================================================== 
Please access the attached hyperlink for an important electronic communications disclaimer:

http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
=============================================================================== 

Mime
View raw message