mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Jambor <jambo...@gmail.com>
Subject Re: out for memory
Date Fri, 18 Jun 2010 14:13:22 GMT
Thanks. In the end, I got away with 7.3GB, I guess this is as good as it 
gets for now.

On 18/06/2010 14:52, Sean Owen wrote:
> GDM needs roughly 28 bytes per preference, all told. I'd expect it
> alone consumes "just" 2.8GB of heap on the Netflix data set. The rest
> may be consumed by SVD matrices and your application storage and other
> JVM stuff.
>
> 10GB seems very large. Are you setting the new generation size to be
> relatively small? otherwise the defaults will probably waste a couple
> gigabytes of heap.
>
> Do what works best for you though.
>
> On Fri, Jun 18, 2010 at 2:41 PM, Tamas Jambor<jamborta@gmail.com>  wrote:
>    
>> I mean I used to run SVD with my implementation storing only three arrays
>> (user, item, rating), that I was able to fit in 3GB memory. I guess
>> GenericDataModel takes up quite a lot of memory, because the data is indexed
>> by users and by items, which is not necessary for SVD
>>
>> On 18/06/2010 14:21, Sean Owen wrote:
>>      
>>> Memory requirements may be much higher for this algorithm as it builds
>>> large intermediate data structures to compute the SVD. Yes I think the
>>> simple data fits in 3GB or so. Sounds like you have solved your
>>> problem by supplying more memory.
>>>
>>>
>>>        
>>      

Mime
View raw message