mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suneel Marthi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAHOUT-1693) FunctionalMatrixView materializes row vectors in scala shell
Date Tue, 21 Apr 2015 03:18:58 GMT

     [ https://issues.apache.org/jira/browse/MAHOUT-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Suneel Marthi updated MAHOUT-1693:
----------------------------------
    Description: 
FunctionalMatrixView materializes row vectors in scala shell.

Problem first reported by a user Michael Alton, Intel:

"When I first tried to make a large matrix, I got an out of Java heap space error. I increased
the memory incrementally until I got it to work. “export MAHOUT_HEAPSIZE=8000” didn’t
work, but “export MAHOUT_HEAPSIZE=64000” did. The question is why do we need so much memory?
A 5000x5000 matrix of doubles should only take up ~200MB of space?"

Problem has been narrowed down to not override toString() method in FunctionalMatrixView which
causes it to materialize all of the row vectors when run in Mahout Spark Shell.

  was:
FunctionalMatrixView materializes row vectors in scala shell.

Problem first reported by Michael Alton, Intel.

"When I first tried to make a large matrix, I got an out of Java heap space error. I increased
the memory incrementally until I got it to work. “export MAHOUT_HEAPSIZE=8000” didn’t
work, but “export MAHOUT_HEAPSIZE=64000” did. The question is why do we need so much memory?
A 5000x5000 matrix of doubles should only take up ~200MB of space?"

Problem has been narrowed down to not override toString() method in FunctionalMatrixView which
causes it to materialize all of the row vectors when run in Mahout Spark Shell.


> FunctionalMatrixView materializes row vectors in scala shell
> ------------------------------------------------------------
>
>                 Key: MAHOUT-1693
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1693
>             Project: Mahout
>          Issue Type: Bug
>          Components: Mahout spark shell, Math
>    Affects Versions: 0.10.0
>            Reporter: Suneel Marthi
>            Assignee: Andrew Palumbo
>            Priority: Blocker
>             Fix For: 0.10.1
>
>
> FunctionalMatrixView materializes row vectors in scala shell.
> Problem first reported by a user Michael Alton, Intel:
> "When I first tried to make a large matrix, I got an out of Java heap space error. I
increased the memory incrementally until I got it to work. “export MAHOUT_HEAPSIZE=8000”
didn’t work, but “export MAHOUT_HEAPSIZE=64000” did. The question is why do we need
so much memory? A 5000x5000 matrix of doubles should only take up ~200MB of space?"
> Problem has been narrowed down to not override toString() method in FunctionalMatrixView
which causes it to materialize all of the row vectors when run in Mahout Spark Shell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message