systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SYSTEMML-2419) Setup and cleanup of remote workers
Date Sat, 07 Jul 2018 05:11:00 GMT

    [ https://issues.apache.org/jira/browse/SYSTEMML-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535629#comment-16535629
] 

Matthias Boehm commented on SYSTEMML-2419:
------------------------------------------

Yes, this can happen if the input sizes to functions are unknown. However, there is a simple
fix. Whenever we force parfor into {{MR}} or {{SPARK}} execution modes, we use {{ParForProgramBlock.checkMRAndRecompileToCP}}
to compile all operations to CP. Maybe you could move this function to a central place (e.g.,
{{Recompiler}}) and call it both from the paramserv and parfor functionality. 

> Setup and cleanup of remote workers
> -----------------------------------
>
>                 Key: SYSTEMML-2419
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2419
>             Project: SystemML
>          Issue Type: Sub-task
>            Reporter: LI Guobao
>            Assignee: LI Guobao
>            Priority: Major
>
> In the context of distributed spark env, we need to firstly ship the necessary functions
and variables to the remote workers and then to initialize and register the cleanup of buffer
pool for each remote worker. All these are inspired by the parfor implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message