hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dyer <rd...@iastate.edu>
Subject Re: Strange machine behavior
Date Tue, 11 Dec 2012 03:30:30 GMT
On Mon, Dec 10, 2012 at 1:23 PM, Andy Isaacson <adi@cloudera.com> wrote:

> What kernel did you see this on? Was there significant swap traffic
> (si/so in vmstat output) during the high-system-time period?
>

It's an older kernel, Fedora 15.

Linux XXXXX 2.6.43.8-1.fc15.x86_64 #1 SMP Mon Jun 4 20:33:44 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux

The next time it happens I'll take a look at the vmstat output, I do not
have that log for this last occurrence.


> BTW, you don't need to nor do you want to run sync(1) when
> manipulating drop_caches, it just causes additional noise and
> slowdown. drop_caches doesn't have any impact on correctness; it won't
> cause data loss (by dropping a dirty page or whatever). I've had sync
> calls take 10 minutes to complete, so the unnecessary impact can be
> significant.
>
> -andy
>
> On Sat, Dec 8, 2012 at 4:09 PM, Robert Dyer <rdyer@iastate.edu> wrote:
> > Has anyone experienced a TaskTracker/DataNode behaving like the attached
> > image?
> >
> > This was during a MR job (which runs often).  Note the extremely high
> System
> > CPU time.  Upon investigating I saw that out of 64GB ram the system had
> > allocated almost 45GB to cache!
> >
> > I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync"
> which is
> > roughly where the graph goes back to normal (much lower System, much
> higher
> > User).
> >
> > This has happened a few times.
> >
> > I have tried playing with the sysctl vm.swappiness value (default of 60)
> by
> > setting it to 30 (which it was at when the graph was collected) and now
> to
> > 10.  I am not sure that helps.
> >
> > Any ideas?  Anyone else run into this before?
> >
> > 24 cores
> > 64GB ram
> > 4x2TB sata3 hdd
> >
> > Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap)
> on
> > this machine.
> >
> > 24 map slots (1gb heap each), no reducers.
> >
> > Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
>



-- 

Robert Dyer
rdyer@iastate.edu

Mime
View raw message