systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (SYSTEMML-1753) OOM on parfor local in-memory result merge
Date Sun, 09 Jul 2017 06:26:00 GMT

     [ https://issues.apache.org/jira/browse/SYSTEMML-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matthias Boehm closed SYSTEMML-1753.
------------------------------------

> OOM on parfor local in-memory result merge
> ------------------------------------------
>
>                 Key: SYSTEMML-1753
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1753
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> Consider a scenario with relatively large parfor result variable (e.g., many iterations
with relatively large output vector each). There are conditions under which the parfor local
in-memory result merge runs unnecessarily out of memory (as shown below) due to allocating
the result in sparse, collecting all (sparse) outputs and finally converting this result to
dense. For result merge without compare, the target number of non-zeros are exactly known,
allowing us to directly allocate the result in the correct format, which reduces memory pressure
by more than 2x (dense matrix in sparse format).
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at org.apache.sysml.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:362)
>         at org.apache.sysml.runtime.matrix.data.MatrixBlock.sparseToDense(MatrixBlock.java:1136)
>         at org.apache.sysml.runtime.matrix.data.MatrixBlock.examSparsity(MatrixBlock.java:1019)
>         at org.apache.sysml.runtime.controlprogram.parfor.ResultMergeLocalMemory.executeSerialMerge(ResultMergeLocalMemory.java:114)
>         at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.consolidateAndCheckResults(ParForProgramBlock.java:1751)
>         at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeLocalParFor(ParForProgramBlock.java:814)
>         at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:635)
>         at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message