systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SYSTEMML-2188) Unnecessary evictions on rdd collect
Date Fri, 16 Mar 2018 04:22:00 GMT
Matthias Boehm created SYSTEMML-2188:
----------------------------------------

             Summary: Unnecessary evictions on rdd collect 
                 Key: SYSTEMML-2188
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2188
             Project: SystemML
          Issue Type: Sub-task
            Reporter: Matthias Boehm


For robustness regarding potential OOMs we already have functionality for guarded collects
that write the RDD to hdfs and read it into memory instead of collect because the latter requires
twice the memory of a simple read. However, there are scenarios, where we collect an RDD and
because its size exceeds the buffer pool, we immediately evict to local file system in a single-threaded
manner. This task aims to consolidate this and use the guarded collect whenever the data is
known to exceed the buffer pool size.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message