systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Created] (SYSTEMML-2172) Repartitioning before caching ulta-sparse matrices
Date Wed, 07 Mar 2018 08:19:00 GMT
Matthias Boehm created SYSTEMML-2172:

             Summary: Repartitioning before caching ulta-sparse matrices
                 Key: SYSTEMML-2172
             Project: SystemML
          Issue Type: Bug
            Reporter: Matthias Boehm

Ultra-sparse matrices have dedicated serialized block representation which means that their
in-memory storage in CSR can be much larger than on disk which leads to a blow-up of 128MB
partitions to >1GB partitions. Accordingly, we should repartition the data before the initial
caching in order to remove memory pressure and exploit the full parallelism.

This message was sent by Atlassian JIRA

View raw message