mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: Scaling question
Date Mon, 14 Mar 2011 01:49:43 GMT
Should they be striped (RAID 0) without duplication? This was the
wisdom I've received.

Before starting each job, run this: "cat inputdirectory/* >
/dev/null". This loads the disk cache with the input files; which
helps a surprising amount.

On Sun, Mar 13, 2011 at 4:45 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> Hadoop doesn't make good use of SSD's.
>
> Just adding more spindles will do more than anything else.
>
> On Sun, Mar 13, 2011 at 5:38 AM, Dave Stuart <
> david.stuart@progressivealliance.co.uk> wrote:
>
>> Thanks Sean,
>>
>> At the moment the disks aren't super fast (7200 rpm) so I was planning on
>> adding some SSD into the mix.
>> Thanks for the tips
>>
>> Regards,
>>
>> Dave
>>
>>
>>
>> On 13 Mar 2011, at 11:20, Sean Owen wrote:
>>
>> > There's no real point in making virtual machines in order to do more
>> > work per machine -- just make Hadoop run more workers per machine. A
>> > good first approximation is indeed to run one worker per core.
>> >
>> > I think you'll find a lot of Mahout-related jobs are I/O-bound, not
>> > CPU-bound. So you may reach a bottleneck with fewer workers than that.
>> > And then you may find you get more bang for your buck not with more
>> > RAM or cores but more and faster disks, and getting Hadoop to use
>> > them.
>> >
>> > On Sun, Mar 13, 2011 at 10:41 AM, David Stuart
>> > <david.stuart@progressivealliance.co.uk> wrote:
>> >> Hey,
>> >>
>> >> I have done my initial tests locally  and now want to building a
>> cluster. My question is currently I have three big machines (32gb ram and 2
>> x 6 cores), would it be more effective/faster keep the machines as is or to
>> divide them into virtual machines and have say 6 machines per Server.
>> >>
>> >> Regards
>> >>
>> >> David Stuart
>>
>>
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message