reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Chung (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1403) Deadlock between ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm)
Date Tue, 24 May 2016 20:28:12 GMT

    [ https://issues.apache.org/jira/browse/REEF-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298827#comment-15298827
] 

Andrew Chung commented on REEF-1403:
------------------------------------

[~MariiaMykhailova] thanks for discovering this!

Taking a brief look at the Java code, this seems like it could happen in Java as well since
the lock is held on the same C# equivalents in Java. {{lock(this)}} is a bad idea in C# because
it is hard to tell who else is calling {{lock}} on the object from the outside\[0\]. I don't
think the problem with the code is whether we are locking on the right objects but rather
that we have a circular dependency on locks:

{{HeartbeatManager->ContextLifeCycle}}, via getting all the context statuses on the heartbeat
loop, and 

{{ContextLifeCycle->HeartbeatManager}}, via updates on the context stack.

In this case it will not matter even if both of them are calling {{lock}} on {{private readonly}}
objects, there is still a circular dependency. To break the dependency, we can either have:

# Both of them try to grab the same lock or 
# Prevent one of them from calling the other.

I'm in favor of the second option, by changing {{HeartbeatManager.OnNext}} to {{private}}
and adding an {{internal}} non-blocking {{QueueHeartBeat}} method to {{HeartbeatManager}}
and have it be triggered on an {{Alarm}}. [~markus.weimer] [~MariiaMykhailova] What do you
think?

\[0\]: http://stackoverflow.com/questions/251391/why-is-lockthis-bad

> Deadlock between ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm)
> ----------------------------------------------------------------------------
>
>                 Key: REEF-1403
>                 URL: https://issues.apache.org/jira/browse/REEF-1403
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF.NET Evaluator
>            Reporter: Mariia Mykhailova
>
> We have a potential deadlock in task start/timed heartbeats.
> {{ContextRuntime.StartTask}} does the following:
> 1. Acquires lock on {{ContextRuntime._contextLifeCycle}}
> 2. Calls {{TaskRuntime.RunTask()}} which calls {{TaskStatus.setInit}} which calls {{HeartBeatManager.onNext(TaskStatusProto)}}
which acquires lock on {{HeartBeatManager}} itself.
> At the same time {{HeartBeatManager.onNext(Alarm)}} gets called every 4 seconds since
evaluator start, and it:
> 1. Acquires lock on {{HeartBeatManager}}.
> 2. Calls {{GetEvaluatorHeartbeatProto}} which calls {{ContextManager.GetTaskStatus()}}
which calls {{ContextManager.GetTaskStatus()}} which acquires lock on {{ContextRuntime._contextLifeCycle}}.
> If task will be starting at the same time as heartbeat kick in, these two actions will
deadlock each other. I encountered a repro on our tests when working on REEF-1388, but in
general case it's not impossible that second or third task that starts on evaluator runs into
a heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message