systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Commented] (SYSTEMML-1879) Parfor remote spark w/ reuse of shared inputs
Date Fri, 01 Sep 2017 08:56:00 GMT


Matthias Boehm commented on SYSTEMML-1879:

FYI [~Tenma] - this will also help for the deep learning scenarios (weights are just read
once per process not once per core, and probably less GC overhead).

> Parfor remote spark w/ reuse of shared inputs
> ---------------------------------------------
>                 Key: SYSTEMML-1879
>                 URL:
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: APIs, Runtime
>            Reporter: Matthias Boehm
>             Fix For: SystemML 1.0
> Currently, we read shared inputs redundantly in each parfor worker. This causes redundant
read and is unnecessarily memory-inefficient.
> This task aims to read shared inputs once per process and reuse them across threads.
The most elegant way of handling this is to reuse initially parsed symbol table entries (instances
of matrix objects), except for result variables. Then the result happens automatically over
the shared per-process buffer pool. 

This message was sent by Atlassian JIRA

View raw message