aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Erb <s...@apache.org>
Subject Review Request 53519: WIP: ExceptionalThread that will tear down the entire application on unhandled errors
Date Sun, 06 Nov 2016 23:26:19 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53519/
-----------------------------------------------------------

Review request for Aurora, John Sirois and Zameer Manji.


Bugs: AURORA-1801
    https://issues.apache.org/jira/browse/AURORA-1801


Repository: aurora


Description
-------

I was not able to manually trigger the root cause of AURORA-1801 by altering the Mesos filesystem
layout. I have therefore adopted the general teardown idea. This is a proof of concept. Feedback
welcome.

Example output (using a hardcoded throw):

```
Bottle v0.11.6 server starting up (using CherryPyServer())...
Listening on http://192.168.33.7:1338/
Hit Ctrl-C to quit.

E1106 23:03:36.722500 8699 exceptional.py:41] Unhandled error in thread Thread-1 [TID=8705].
Tearing down.
Traceback (most recent call last):
  File "apache/thermos/common/exceptional.py", line 37, in _excepting_run
    self.__real_run(*args, **kw)
  File "apache/thermos/observer/task_observer.py", line 135, in run
    self._detector.refresh()
  File "apache/thermos/observer/detector.py", line 74, in refresh
    self._refresh_detectors()
  File "apache/thermos/observer/detector.py", line 58, in _refresh_detectors
    new_paths = set(self._path_detector.get_paths())
  File "apache/aurora/executor/common/path_detector.py", line 35, in get_paths
    return list(set(path for path in iterate() if os.path.exists(path)))
  File "apache/aurora/executor/common/path_detector.py", line 35, in <genexpr>
    return list(set(path for path in iterate() if os.path.exists(path)))
  File "apache/aurora/executor/common/path_detector.py", line 34, in iterate
    raise RuntimeError("Fail on purpose...")
RuntimeError: Fail on purpose...
I1106 23:03:42.513900 8728 static_assets.py:34] detecting assets...
I1106 23:03:42.541809 8728 static_assets.py:38]   detected asset: observer.js
I1106 23:03:42.542799 8728 static_assets.py:38]   detected asset: bootstrap.css
I1106 23:03:42.543728 8728 static_assets.py:38]   detected asset: jquery.pailer.js
I1106 23:03:42.544576 8728 static_assets.py:38]   detected asset: jquery.js
I1106 23:03:42.548482 8728 static_assets.py:38]   detected asset: favicon.ico
Bottle v0.11.6 server starting up (using CherryPyServer())...
Listening on http://192.168.33.7:1338/
Hit Ctrl-C to quit.
```

Questions:

* Is it a good idea to solve the teardown problem like that?
* Do we want to update all current usages of `twitter.common.exceptions.ExceptionalThread`?
* Where do we put that code so that we can use it in client, executor and observer?


Diffs
-----

  src/main/python/apache/aurora/tools/thermos_observer.py 0a07df7bed902be2b73d2ca3308a02a1e34760d6

  src/main/python/apache/thermos/common/BUILD 0adabbfa3f0230dae95aca5002c128832916fdd6 
  src/main/python/apache/thermos/common/exceptional.py PRE-CREATION 
  src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427


Diff: https://reviews.apache.org/r/53519/diff/


Testing
-------

./pants test.pytest src/{main,test}/python:: -- -v
aurorabuild observer


Thanks,

Stephan Erb


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message