Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates
 200.55.140.180 as permitted sender)
Message-ID: <50C3E3DC.3000709@uci.cu>
Date: Sat, 08 Dec 2012 20:05:32 -0500
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:16.0) Gecko/20121028 Thunderbird/16.0.2
MIME-Version: 1.0
To: user@hadoop.apache.org
CC: Robert Dyer <rdyer@iastate.edu>
Subject: Re: Strange machine behavior
References: 
 <CAMiz3FMkKRKciK=iP9TpqecbyE50uM4QdfXOTObq=UACfmiJvA@mail.gmail.com>
In-Reply-To: 
 <CAMiz3FMkKRKciK=iP9TpqecbyE50uM4QdfXOTObq=UACfmiJvA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Are you sure that 24 map slots is a good number for this machine?
Remember that you have three services (DN, TT and HRegionServer) with
with a 12 GB for Heap.
Try to use a lower number of map slots (12 for example) and launch your
MR job again.
Can you share your logs in pastebin?


On Sat 08 Dec 2012 07:09:02 PM CST, Robert Dyer wrote:
> Has anyone experienced a TaskTracker/DataNode behaving like the
> attached image?
>
> This was during a MR job (which runs often).  Note the extremely high
> System CPU time.  Upon investigating I saw that out of 64GB ram the
> system had allocated almost 45GB to cache!
>
> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync"
> which is roughly where the graph goes back to normal (much lower
> System, much higher User).
>
> This has happened a few times.
>
> I have tried playing with the sysctl vm.swappiness value (default of
> 60) by setting it to 30 (which it was at when the graph was collected)
> and now to 10.  I am not sure that helps.
>
> Any ideas?  Anyone else run into this before?
>
> 24 cores
> 64GB ram
> 4x2TB sata3 hdd
>
> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb
> heap) on this machine.
>
> 24 map slots (1gb heap each), no reducers.
>
> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.

--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>


10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci