aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zameer Manji <zma...@apache.org>
Subject Re: Review Request 51876: Modify executor state transition logic to rely on health checks (if enabled)
Date Mon, 26 Sep 2016 17:41:04 GMT


> On Sept. 23, 2016, 3:18 p.m., Zameer Manji wrote:
> > src/main/python/apache/aurora/executor/aurora_executor.py, line 120
> > <https://reviews.apache.org/r/51876/diff/5/?file=1509363#file1509363line120>
> >
> >     This check is brittle to determine if health checking is enabled. Please consider
an alternative approach.
> >     
> >     You have access to the `assigned_task` object. Passing that in to the helper
method `mesos_task_instance_from_assigned_task` will give you an instance of the executor
config.
> >     
> >     You can then check the `health_check_config()` property of the result to see
if health checking is enabled for the task.
> 
> Kai Huang wrote:
>     Seems like we are taking a step back? In the first revision, I implemented pretty
much the way you mentioned above, that is creating a is_health_check_enabled(assigned_task)
function in task_info.py. 
>     
>     However, Maxim raised a valid point that is_health_check_enabled has some duplication
with the creation of health checker in later step.
>     So the problem would be whether we should reuse the logic of is_health_check_enabled
in health_checker?
>     
>     One solution is to store all the computation result(prot_map, health_checker, health_check_config)
in a utility class. So that it can be reuse later. But a downside here is that the is_health_check_enabled
now serves multiple purposes, and the meaning of this function is not clear. It should only
answer one question: is health check enabled on this task? 
>     
>     From my perspective, I think we should allow some sacrifice of reusability here.

You make a good argument here, so I will drop my objection.


- Zameer


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51876/#review150246
-----------------------------------------------------------


On Sept. 23, 2016, 11:58 a.m., Kai Huang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51876/
> -----------------------------------------------------------
> 
> (Updated Sept. 23, 2016, 11:58 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji.
> 
> 
> Bugs: AURORA-1225
>     https://issues.apache.org/jira/browse/AURORA-1225
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Modify executor state transition logic to rely on health checks (if enabled).
> 
> [Summary]
> Executor needs to start executing user content in STARTING and transition to RUNNING
when a successful required number of health checks is reached.
> 
> This review contains a series of executor changes that implement the health check driven
updates. It gives more context of the design of this feature.
> 
> [Background]
> Please see this epic: https://issues.apache.org/jira/browse/AURORA-1225
> and the design doc: https://docs.google.com/document/d/1ZdgW8S4xMhvKW7iQUX99xZm10NXSxEWR0a-21FP5d94/edit#
for more details and background.
> 
> [Description]
> If health check is enabled on vCurrent executor, the health checker will send a "TASK_RUNNING"
message when a successful required number of health checks is reached within the initial_interval_secs.
On the other hand, a "TASK_FAILED" message was sent if the health checker fails to reach the
required number of health checks within that period, or a maximum number of failed health
check limit is reached after the initital_interval_secs.
> 
> If health check is disabled on the vCurrent executor, it will sends "TASK_RUNNING" message
to scheduler after the thermos runner was started. In this scenario, the behavior of vCurrent
executor will be the same as the vPrev executor.
> 
> [Change List]
> The current change set includes:
> 1. Removed the status memoization in ChainedStatusChecker.
> 2. Modified the StatusManager to be edge triggered.
> 3. Changed the Aurora Executor callback function.
> 4. Modified the Health Checker and redefined the meaning initial_interval_secs.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/executor/aurora_executor.py ce5ef680f01831cd89fced8969ae3246c7f60cfd

>   src/main/python/apache/aurora/executor/common/health_checker.py 5fc845eceac6f0c048d7489fdc4c672b0c609ea0

>   src/main/python/apache/aurora/executor/common/status_checker.py 795dae2d6b661fc528d952c2315196d94127961f

>   src/main/python/apache/aurora/executor/status_manager.py 228a99a05f339e21cd7e769a42b9b2276e7bc3fc

>   src/test/python/apache/aurora/executor/common/test_health_checker.py bb6ea69dd94298c5b8cf4d5f06d06eea7790d66e

>   src/test/python/apache/aurora/executor/common/test_status_checker.py 5be1981c8c8e88258456adb21aa3ca7c0aa472a7

>   src/test/python/apache/aurora/executor/test_status_manager.py ce4679ba1aa7b42cf0115c943d84663030182d23

>   src/test/python/apache/aurora/executor/test_thermos_executor.py 0bfe9e931f873c9f804f2ba4012e050e1f9fd24e

> 
> Diff: https://reviews.apache.org/r/51876/diff/
> 
> 
> Testing
> -------
> 
> ./build-support/jenkins/build.sh
> 
> ./pants test.pytest src/test/python/apache/aurora/executor::
> 
> 
> Thanks,
> 
> Kai Huang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message