systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Commented] (SYSTEMML-2420) Communication between ps and workers
Date Mon, 25 Jun 2018 05:37:00 GMT


Matthias Boehm commented on SYSTEMML-2420:

Yes, in general this is a good start. It would be awesome to reuse Spark's {{RpcEnv}} because
it would automatically and consistently use the spark configurations {{spark.rpc.*}} along
with potential changes of default values across versions. However, if this creates problems
(i.e., if it is difficult to realize), we can simply use Netty directly. If I remember correctly
that's how Spark also implements its RPC communication.

> Communication between ps and workers
> ------------------------------------
>                 Key: SYSTEMML-2420
>                 URL:
>             Project: SystemML
>          Issue Type: Sub-task
>            Reporter: LI Guobao
>            Assignee: LI Guobao
>            Priority: Major
> It aims to implement the parameter exchange between ps and workers. We could leverage
spark RPC to setup a ps endpoint in driver node which means that the ps service could be discovered
by workers in the network. And then the workers could invoke the pull/push method via RPC
using the registered endpoint of ps service. Hence, in details, this tasks consists of registering
the ps endpoint in spark rpc framework and using rpc to invoke target method in worker side.
We can learn that the spark rpc is implemented in Scala. Hence we need to wrap them in in
order to be used in Java. Overall, we could register the ps service with _RpcEndpoint_ and
invoke the service with _RpcEndpointRef_.

This message was sent by Atlassian JIRA

View raw message