reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Service <dsop...@gmail.com>
Subject Which network classes to use on C# side
Date Thu, 06 Apr 2017 19:23:09 GMT
This week I am working on an example REEF application with Scott Inglis
that uses REEF to run R applications in parallel. The R language itself is
single threaded, but there is an R package called foreach that provides an
interface to run each iteration of a foreach loop in parallel. The foreach
package requires another backend package to implement the parallelism and a
number of these packages exist, doSNOW, doMPI, doParallel. Without a
backend, foreach just runs the computations sequentially. REEF is an
interesting platform to on which to implement a foreach backend because it
also provides the ability to rerun failed tasks.

We have implemented the REEF application, driver, and tasks in C#, and
currently have an R script being sent from the driver to the tasks where R
is spawned, the script is run and the results are returned to the driver.

Questions:
I need to provide message transport between the application and the driver.
The suspend/resume Java example uses the Transport interface. Which
REEF.Network or REEF.Wake C# class is most similar to this one or what
would you recommend (TransportClient/TransportServer)? Is there a good
example on the C# side of application/driver two way messaging?

BTW
I am seeing asymmetries between the C# and Java side. For example, the Java
DriverConfiguration has ON_CLIENT_MESSAGE and an equivalent does not exist
on the C# side.Is there a reason for this and it would be much work to
implement? It seems to me that messaging between the application/client and
driver but fundamental to REEF and easy to configure.

The Java side also has a ClientConfiguration, but the C# side only  has
LocalRuntimeClientConfiguration, YARNClientConfiguration, etc but no
ClientConfiguration.

Doug

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message