systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fei Hu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SYSTEMML-1831) Improve the efficiency of matrix subsetting
Date Tue, 08 Aug 2017 18:57:00 GMT

     [ https://issues.apache.org/jira/browse/SYSTEMML-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Fei Hu updated SYSTEMML-1831:
-----------------------------
    Description: 
For the {{rangeReIndex}} operation, it needs to read the whole input matrix into memory first
and do the subsetting in the memory. It is not efficient because it needs to read a lot of
unnecessary data, and even will be out of memory if the size of input matrix exceeds the available
memory. 

The plan here is to read the keys in the Hadoop sequence file, and identify the keys overlapped
with the input range index. Then the values specified by the identified keys will be read.

  was:
For the {{rangeReIndex}} operation, it needs to read the whole input matrix into memory first
and do the subsetting in the memory. It is not efficient because it needs to read a lot of
unnecessary data, and even out of memory if the size of input matrix exceeds the available
memory. 

The plan here is to read the keys in the Hadoop sequence file, and identify the keys overlapped
with the input range index. Then the values specified by the identified keys will be read.


> Improve the efficiency of matrix subsetting
> -------------------------------------------
>
>                 Key: SYSTEMML-1831
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1831
>             Project: SystemML
>          Issue Type: Improvement
>            Reporter: Fei Hu
>            Assignee: Fei Hu
>
> For the {{rangeReIndex}} operation, it needs to read the whole input matrix into memory
first and do the subsetting in the memory. It is not efficient because it needs to read a
lot of unnecessary data, and even will be out of memory if the size of input matrix exceeds
the available memory. 
> The plan here is to read the keys in the Hadoop sequence file, and identify the keys
overlapped with the input range index. Then the values specified by the identified keys will
be read.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message