systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Berthold Reinwald (JIRA)" <>
Subject [jira] [Updated] (SYSTEMML-1349) Parfor optimizer support for shared reads (lower memory req)
Date Thu, 21 Dec 2017 06:05:05 GMT


Berthold Reinwald updated SYSTEMML-1349:
    Fix Version/s:     (was: SystemML 1.0)
                   SystemML 1.1

> Parfor optimizer support for shared reads (lower memory req)
> ------------------------------------------------------------
>                 Key: SYSTEMML-1349
>                 URL:
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: Compiler
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.1
> Let's assume the following example script and focus on local parfor with a memory budget
of 10GB and k=16 threads (but this issue similarly applies to parfor spark).
> {code}
> X = rand(rows=1000000, cols=1000); #8GB
> parfor(i in 1:ncol(X)) 
>     print(sum(X[,i]));
> {code}
> The memory estimate of the right indexing operation is 8GB and the parfor optimizer tries
to set k=min(floor(10GB/8GB), 16) and hence falls back to a single-threaded plan with k=1.
However, this is completely unnecessary because X is a shared-read (i.e., variable that is
never updated in the parfor body). Accordingly, we should determine the degree of parallelism
as k=min(floor((10GB-8GB)/8MB), 16).
> The challenge is to do this for arbitrary control flow and multiple shared reads of different
dimension and sparsity. 

This message was sent by Atlassian JIRA

View raw message