hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: What resources are used by idle NameNode and JobTracker tasks
Date Wed, 13 Feb 2013 17:41:37 GMT
The NN has a constant memory use and the CPU usage should also be
patterned in idle mode. The JT usually retires jobs over the days, and
the memory and CPU usage should both go down over time in idle mode,
especially if you indicate a week of idleness. There is however the DN
block scanner that may trigger itself and begin consuming a bit of an
IO in verifying block data integrity (to re-populate rot/etc.
corrupted blocks on disks), which would probably be the only usage
registered on an idle cluster with no users talking to it.

What exact 'resources' was the complaint about? Having more idea on
that would help us trace or answer better.

On Mon, Feb 11, 2013 at 11:37 PM, Steve Lewis <lordjoe2000@gmail.com> wrote:
> I have Hadoop running on 8 nodes in a 50 node cluster. The NameNode and
> Jobtrackers have been running for months while jobs were run occasionally -
> usually no more than a dozen a week and more typically none. Recently the
> cluster manager accused Hadoop of consuming a large number of resources in
> the cluster - this occurred during a week an which I was running no jobs.
> While I stopped Hadoop  the trackers have been running for many months
> stably.
>   Is there any reason why either of these jobs should suddenly and without
> jobs being run increase their consumption of resources in a serious way??
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com

Harsh J

View raw message