hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bharath Mundlapudi <bharathw...@yahoo.com>
Subject Re: Strange machine behavior
Date Tue, 18 Dec 2012 08:42:19 GMT
You may want to check why System time is high. Check your system call stats. This should give
you some clue.


 From: Robert Dyer <rdyer@iastate.edu>
To: user@hadoop.apache.org; Bharath Mundlapudi <bharathwork@yahoo.com> 
Sent: Monday, December 10, 2012 7:32 PM
Subject: Re: Strange machine behavior

Yes there is performance impact.  It should be visible from the graph I attached.  Basically,
the CPU is spending much more time on System and the User time is lowered.

When this happens (if I don't do a drop_caches in time) the MR job winds up taking significantly
longer than usual.

On Mon, Dec 10, 2012 at 8:06 PM, Bharath Mundlapudi <bharathwork@yahoo.com> wrote:

Are you seeing any performance impact with this cache increase? It is normal in linux system
to grab high cache level. 
> From: Andy Isaacson <adi@cloudera.com>
>To: user@hadoop.apache.org 
>Sent: Monday, December 10, 2012 11:23 AM
>Subject: Re: Strange machine behavior
>What kernel did you see this on? Was there significant swap traffic
>(si/so in vmstat output) during the high-system-time period?
>BTW, you don't need to nor do you want to run sync(1) when
>manipulating drop_caches, it just causes additional noise and
>slowdown. drop_caches doesn't have any impact on correctness; it won't
>cause data loss (by dropping a dirty page or whatever). I've had sync
>calls take 10 minutes to complete, so the unnecessary impact can be
>On Sat, Dec 8, 2012 at 4:09 PM, Robert Dyer <rdyer@iastate.edu> wrote:
>> Has anyone experienced a TaskTracker/DataNode behaving like the attached
>> image?
>> This was during a MR job (which runs often).  Note the extremely high System
>> CPU time.  Upon investigating I saw that out of 64GB ram the system had
>> allocated
 almost 45GB to cache!
>> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" which is
>> roughly where the graph goes back to normal (much lower System, much higher
>> User).
>> This has happened a few times.
>> I have tried playing with the sysctl vm.swappiness value (default of 60) by
>> setting it to 30 (which it was at when the graph was collected) and now to
>> 10.  I am not sure that helps.
>> Any ideas?  Anyone else run into this before?
>> 24 cores
>> 64GB ram
>> 4x2TB sata3 hdd
>> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap) on
>> this machine.
>> 24 map slots (1gb heap each), no reducers.
>> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.


Robert Dyer
View raw message