reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1403) Deadlock between ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm)
Date Tue, 24 May 2016 19:43:14 GMT

    [ https://issues.apache.org/jira/browse/REEF-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298772#comment-15298772
] 

Markus Weimer commented on REEF-1403:
-------------------------------------

I believe the way we avoided that on the Java side is to use (1) a queue for heartbeats and
(2) a global lock for Evaluator state transitions. The queue makes sure you send the heartbeats
in order. The global lock makes sure that there aren't two heartbeats being assembled at the
same time. We actually use the {{HeartbeatManager}}'s singleton instance for that lock, and
mark all its methods as {{synchronized}}, which is the equivalent of {{lock(this)}} in C#.
[~afchung90] explained to me offline why that is a bad idea in C#, but I forgot :-(

> Deadlock between ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm)
> ----------------------------------------------------------------------------
>
>                 Key: REEF-1403
>                 URL: https://issues.apache.org/jira/browse/REEF-1403
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF.NET Evaluator
>            Reporter: Mariia Mykhailova
>
> We have a potential deadlock in task start/timed heartbeats.
> {{ContextRuntime.StartTask}} does the following:
> 1. Acquires lock on {{ContextRuntime._contextLifeCycle}}
> 2. Calls {{TaskRuntime.RunTask()}} which calls {{TaskStatus.setInit}} which calls {{HeartBeatManager.onNext(TaskStatusProto)}}
which acquires lock on {{HeartBeatManager}} itself.
> At the same time {{HeartBeatManager.onNext(Alarm)}} gets called every 4 seconds since
evaluator start, and it:
> 1. Acquires lock on {{HeartBeatManager}}.
> 2. Calls {{GetEvaluatorHeartbeatProto}} which calls {{ContextManager.GetTaskStatus()}}
which calls {{ContextManager.GetTaskStatus()}} which acquires lock on {{ContextRuntime._contextLifeCycle}}.
> If task will be starting at the same time as heartbeat kick in, these two actions will
deadlock each other. I encountered a repro on our tests when working on REEF-1388, but in
general case it's not impossible that second or third task that starts on evaluator runs into
a heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message