systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SYSTEMML-1423) OOM on generating ultra-sparse rand data
Date Tue, 02 May 2017 18:26:04 GMT

     [ https://issues.apache.org/jira/browse/SYSTEMML-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Glenn Weidner updated SYSTEMML-1423:
------------------------------------
    Fix Version/s:     (was: SystemML 1.0)
                   SystemML 0.14

> OOM on generating ultra-sparse rand data
> ----------------------------------------
>
>                 Key: SYSTEMML-1423
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1423
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 0.14
>
>
> In order to ensure consistency across backends, we first determine the number of non-zeros
per block and subsequently generate random data accordingly. However, in case of ultra-sparse
data sets, this temporary array can be almost as large as the dataset. Since this memory consumption
is unaccounted and even required for distributed operations, there are various possible scenarios
where this would cause OOMs. 
> This task aims to solve this issue for all backends, by determining the nnz per block
in a streaming manner without materialization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message