Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mp2893@gmail.com designates
 209.85.216.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Cb0z0mSU8Y5vPkrfVoxY48gn+nXJ1HvVG1rlku+4bOinMAe/jolC+/baBJ7xIigcpF
         ZTavLZY9DGtsocIFKyi0DIGTXhghNGP0O6RGczUp3KuSioX+MJFnRnvNHN9SKWd2zviR
         fR5DjYa8aJiwszCd3AmLZ33CIwEvW7GLlxsgI=
MIME-Version: 1.0
Date: Fri, 9 Jul 2010 11:31:48 +0900
Message-ID: <AANLkTilc9gCCReEVfxltb-hpVXJbMm8dJnchZseWb7hi@mail.gmail.com>
Subject: Is "heap size allocation" of namenode dynamic or static?
From: edward choi <mp2893@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000feaea1f3d76c5a9048aeb346c

--000feaea1f3d76c5a9048aeb346c
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Machines in my cluster have relatively small physical memory (4GB)

I was wondering if I could reduce the heap size that namenode and jobtracker
are assigned.
The default heap size is 1000MB respectively, and I know that.
The thing is, does that 1000MB mean maximum possible memory that namenode(or
jobtracker) can use?
What I mean is that does namenode start with minimum memory and increase the
memory size all the way up to 1000MB depending on the job status?
Or is namenode given 1000MB from the beginning so that there is no
flexibility at all?

If namenode and jobtracker do start with solid 1000MB then I would have to
dial them down to several hundreds of mega byte since I only 4GB of memory.
2giga bytes of memory taken up just by namenode and jobtracker is too much
an expense for me.

My question also applies to heap size of child JVM. I know that they are
originally given 200MB of heap size.
I intend to increase the heap size to 512MB, but if the heap size allocation
has no flexibility then I'd have to maintain the 200MB configuration.
Take out the 2GB (used by namenode and jobtracker) from the total 4GB, I can
have only 4 map/reduce tasks with 512MB configuration and since I have quad
core CPU this would be a waste.

Oh, and one last thing.
I am using Hadoop streaming.
I read from a book that when you are using hadoop streaming, you should
allocate less heap size to child JVM. (I am not sure if it meant less than
200MB or less than 400MB)
Because streaming does not allow enough memory for user's processes to run.
So what is the optimal heap size for map/reduce tasks in hadoop streaming?
My plan was to increase the heap size of the child JVM to 512MB.
But if what the book says is true, there is no point.

--000feaea1f3d76c5a9048aeb346c--