reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Service (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (REEF-1763) Replace interop call to StartHandlerOnNext() between Java/C# bridges
Date Mon, 05 Jun 2017 20:39:04 GMT

    [ https://issues.apache.org/jira/browse/REEF-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037538#comment-16037538
] 

Doug Service edited comment on REEF-1763 at 6/5/17 8:38 PM:
------------------------------------------------------------

Answer to comment one: 

Yes the goal is to replace all of the interop calls with Avro messages. There was a consensus
at the time this was proposed that by providing a well defined Avro interface it would allow
us to support bridges in a number of different languages.

I am also concerned about the performance impact of sending the messages over the network.
The existing bridge design seems to rely on the synchronous nature of the interop calls. If
the interface was asynchronous or at least partially asynchronous, the throughput could still
be high even though any one given message takes longer to complete since multiple messages
can be in flight at the same time.

The current pull request specifically addresses the fact that lot of messages will be needed
to complete the work by providing a framework where all messages in org.apache.reef.bridge.message
package are  discovered at system startup using refection on both the C# and Java sides and
then serializers and deserializers are programatically constructed. This means that replacing
a given interop call requires defining the associated messages and implementing the calls
on the java and C# sides as the bridge network will automatically build what is needed to
send and receive the new message. 


was (Author: dougservice):
Answer to comment one: 

Yes the goal is to replace all of the interop calls with Avro messages. There was a consensus
at the time this was proposed that by providing a well defined Avro interface it would allow
us to support bridges in a number of different languages.

I am also concerned about the performance impact of sending the messages over the network.
The existing bridge design seems to rely on the synchronous nature of the interop calls. If
the interface was asynchronous or at least partially asynchronous, the throughput could still
be high even though any one given message takes longer to complete since multiple messages
can be in flight at the same time.

The current pull request specifically addresses the fact that lot of messages will be needed
to complete the work by providing a framework where all messages in org.apache.reef.bridge.message
are  discovered at system startup using refection on both the C# and Java sides and then serializers
and deserializers are programatically constructed. This means that replacing a given interop
call requires defining the associated messages and implementing the calls on the java and
C# sides as the bridge network will automatically build what is needed to send and receive
the new message. 

> Replace interop call to StartHandlerOnNext() between Java/C# bridges
> --------------------------------------------------------------------
>
>                 Key: REEF-1763
>                 URL: https://issues.apache.org/jira/browse/REEF-1763
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF Bridge, REEF Driver
>            Reporter: Doug Service
>            Assignee: Doug Service
>
> Currently bridge communication from Java to C# is performed via managed C++ interop which
is not supported on Linux. This is the first of a number of changes to replace all of the
interop calls between Java and C# with Avro messages carried by the REEF Wake networking classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message