systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Commented] (SYSTEMML-2419) Setup and cleanup of remote workers
Date Fri, 13 Jul 2018 20:47:00 GMT


Matthias Boehm commented on SYSTEMML-2419:

Indeed we usually run Spark with 1 executor per node and k cores per executor, which means
that k tasks will run concurrently in the same executor JVM. I think it would be a natural
first step to similarly run k paramserv workers per executor - this will happen automatically
if you use the current {{foreach}} approach.

Code generation is an advanced feature (by default still disabled) that automatically fuses
multiple operators into composite operators by generating the respective source code, as well
as compiling and loading the respective classes. To ensure robustness with regarding to heterogeneous
JVMs in a cluster, we actually ship the source code and compile the classes at each executor.
The classes are compiled and loaded and subsequently shared across tasks (concurrent and subsequent)
via a basic class cache. 

> Setup and cleanup of remote workers
> -----------------------------------
>                 Key: SYSTEMML-2419
>                 URL:
>             Project: SystemML
>          Issue Type: Sub-task
>            Reporter: LI Guobao
>            Assignee: LI Guobao
>            Priority: Major
> In the context of distributed spark env, we need to firstly ship the necessary functions
and variables to the remote workers and then to initialize and register the cleanup of buffer
pool for each remote worker. All these are inspired by the parfor implementation.

This message was sent by Atlassian JIRA

View raw message