Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of jamborta@gmail.com designates
 74.125.82.170 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=Bb1NPZ4PFVALCBfeRlhWJTxyNoUCVdXKLlZhIiJUd8ZsvWJ/yEwP+NdWz3LTLFN39i
         C9Ddumx8fPUDKCAnXAsNn6PZqlBiRH86Am7Kp4sU5tElX0K7IxnBA3XATXwPlUqdrwDO
         sKkGLDn6VUStFP9MsoZepiZsqzly7B0IvHyUI=
Message-ID: <4C1B7F02.4020806@gmail.com>
Date: Fri, 18 Jun 2010 15:13:22 +0100
From: Tamas Jambor <jamborta@gmail.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB;
 rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1
MIME-Version: 1.0
To: user@mahout.apache.org
Subject: Re: out for memory
References: <1276271105796-888907.post@n3.nabble.com>
	<AANLkTikfkERFHwxalkmYp_k_Ka-N1tOh3itAKbuaUZMX@mail.gmail.com>
	<1276271586879-888926.post@n3.nabble.com>
	<1276273162484-889014.post@n3.nabble.com>
	<AANLkTikdFGGDe_ep9M7yRRsHFZW7lZeBevheEOh4zdnV@mail.gmail.com>
	<1276774920737-902465.post@n3.nabble.com>
	<AANLkTikudr4Lg82v2fhLkiT0-gTebpFsemUV6d_yawpY@mail.gmail.com>
	<1276776558358-902524.post@n3.nabble.com>
	<20100617142106.3e803030@essen.neofonie.priv>
	<AANLkTilleqxd-FNB7HtqkpNa7XkwEXJtulA33gLX7rof@mail.gmail.com>
	<1276851856989-905163.post@n3.nabble.com>
	<AANLkTimkzpMjJs5jCCfeZsr8itS8W41VENG1ag8Ab3p3@mail.gmail.com>
	<4C1B4B6A.9080101@gmail.com>
	<AANLkTilp0XqUF1TAWPlDuNkcynL-7thcq1ZQTMXktIsF@mail.gmail.com>
	<4C1B7059.3030203@gmail.com>
	<AANLkTimFc2DvX4sXSgCEbuUXbZeCumnqwGoeyuBgxXaq@mail.gmail.com>
	<4C1B777D.70500@gmail.com>
 <AANLkTin6VRSeJhLZEu6tu00UwWQjXnX-84rixDI96er8@mail.gmail.com>
In-Reply-To: <AANLkTin6VRSeJhLZEu6tu00UwWQjXnX-84rixDI96er8@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Thanks. In the end, I got away with 7.3GB, I guess this is as good as it 
gets for now.

On 18/06/2010 14:52, Sean Owen wrote:
> GDM needs roughly 28 bytes per preference, all told. I'd expect it
> alone consumes "just" 2.8GB of heap on the Netflix data set. The rest
> may be consumed by SVD matrices and your application storage and other
> JVM stuff.
>
> 10GB seems very large. Are you setting the new generation size to be
> relatively small? otherwise the defaults will probably waste a couple
> gigabytes of heap.
>
> Do what works best for you though.
>
> On Fri, Jun 18, 2010 at 2:41 PM, Tamas Jambor<jamborta@gmail.com>  wrote:
>    
>> I mean I used to run SVD with my implementation storing only three arrays
>> (user, item, rating), that I was able to fit in 3GB memory. I guess
>> GenericDataModel takes up quite a lot of memory, because the data is indexed
>> by users and by items, which is not necessary for SVD
>>
>> On 18/06/2010 14:21, Sean Owen wrote:
>>      
>>> Memory requirements may be much higher for this algorithm as it builds
>>> large intermediate data structures to compute the SVD. Yes I think the
>>> simple data fits in 3GB or so. Sounds like you have solved your
>>> problem by supplying more memory.
>>>
>>>
>>>        
>>