reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Weimer <mar...@weimo.de>
Subject Re: Communication in REEF Wake .NET
Date Sun, 15 Apr 2018 16:31:37 GMT
On Thu, Apr 12, 2018 at 12:31 PM, Chenxi Zhao
<Chenxi.Zhao@microsoft.com.invalid> wrote:
> As far as I get, REEF Wake .NET is the mirror implementation of Java side.

Almost. In Java, Wake serves two purposes: (1) Local event based
programming and (2) event remoting. In C#, we have the Rx extensions
as a local event base programming model. Hence, that part of Wake/Java
has no correspondence in Wake.NET. You can think of Wake.NET as Rx
event remoting designed to be compatible with Wake/Java.

>   1.  I am not seeing where Wake .NET is used for Driver-Client communication, which
differs the implementation from Java side.

That part was never implemented. Even on the Java side, that design is
questionable in the absence of HTTP support in Wake. The Client needs
to be able to disconnect from the Driver (e.g. because you want to
shutdown your laptop) and reconnect later. Further, direct
communication between a machine outside and one machine inside a
cluster / cloud is frequently prohibited by firewalls.

Because of this, the dominant strategy for REEF/Java and REEF.NET is
to launch a HTTP endpoint in the Driver. In fact, this always happens
in REEF.NET. The Client then discovers that endpoint via some
cluster-specific mechanism to establish communications. In the case of
Apache Hadoop YARN, this is done via the HTTP redirection service.
Given a Job ID, one can ask the RM to redirect HTTP calls to the
Driver.

> Q: Is this working in progress?

Kinda. The basic mechanism works, but work remains to be done: The
Java API supports things like sending messages from Driver to Client,
which we don't have a standard way for via the HTTP approach in .NET.
But to the best of my knowledge, nobody is actively working in this
right now.


>   1.  RemoteManager seems to be used in evaluator / driver communication, from code :
https://github.com/apache/reef/blob/master/lang/cs/Org.Apache.REEF.Common/Runtime/Evaluator/EvaluatorRuntime.cs#L72,
it registers the Observer.

Yes. The .NET Evaluator connects to the Java-side of the Driver via Wake.

> Q: I am not seeing where NetworkTransport is used, by searching NetworkTransport through
the project. And I also noticed there is NetworkTransport in Java side as well. But no classes
refers to it as well. How does it work through bridge?

I am no expert on the `NetworkTransport` class. However, the Bridge
doesn't use Wake at all. Instead, the .NET Evaluator connects directly
to the Java-side of the Driver. And that Java side of the Driver then
relays the messages via native code to the .NET Driver running in the
same process. That being said, Doug / Scott / Tyson are hard at work
replacing that with a two-process solution.

>   1.  reef-bridge-client-0.17.0-SNAPSHOT-shaded.jar still contains org\apache\reef\wake
package.
>
> Q: I may misunderstand the overall design on .NET side. Why is Java Wake package needed
in Bridge jar?

The Bridge has a Java side and a .NET side. The Java side needs all of
REEF/Wake/Tang to work.

Hope this helps,

Markus

Mime
View raw message