Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of cnweike@gmail.com designates
 209.85.160.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=XcOkllc2u3S+bXV5wEpjKjzuVt3fDFbC2Qp3UnRvWKe9crZS5bFx2aWaUq3N8ly+Cc
         2B3Ng+Z9KLNfMCdYjkrbYRmCa5W3zT0tz22GstmYpq7KV/eqzD4BkhHlvkkpSsX7L0NJ
         KtRh+hiQ7yYSg3Ffwib0GJHQtjtNZNe7vxQEA=
MIME-Version: 1.0
Date: Tue, 3 May 2011 11:20:55 +0800
Message-ID: <BANLkTik9VHGBrigXUzqPFPMW-zWk62_BMg@mail.gmail.com>
Subject: how to take full advantage of large amount of memory ?
From: Fei Pan <cnweike@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/mixed; boundary=bcaec5215f27d47b8a04a256a05c

--bcaec5215f27d47b8a04a256a05c
Content-Type: multipart/alternative; boundary=bcaec5215f27d47b8404a256a05a

--bcaec5215f27d47b8404a256a05a
Content-Type: text/plain; charset=UTF-8

I am managing a cluster composed of 1 namenode and 8 datanodes (CDH3
hadoop-0.20.2);  they are all 16-core CPU and 32G-memory.

for the full usage of the large memory I configured as following:

 *HADOOP_HEAPSIZE = 1GB (hadoop-env.sh)*
* mapred.child.java.opts =  -Xmx1024m (mapred-site.xml)*
* io.sort.mb = 256M (mapred-site.xml)
*
I want to know *whether these configurations are suitable (should they be
configured smaller or bigger ?) for the dataNode machines*, and *is there
any other configurations which can take more advantage of the  large memory*
;

thank you for your reply in advance.


-- 
Stay Hungry. Stay Foolish.

--bcaec5215f27d47b8404a256a05a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I am managing a cluster composed of 1 namenode and 8 datanodes (CDH3 hadoop=
-0.20.2); =C2=A0they are all 16-core CPU and 32G-memory.=C2=A0<div><br></di=
v><div>for the full usage of the large memory I configured as following:</d=
iv><div>
=C2=A0</div><div>=C2=A0<b>HADOOP_HEAPSIZE =3D 1GB (hadoop-env.sh)</b></div>=
<div><b>=C2=A0mapred.child.java.opts =3D =C2=A0-Xmx1024m (mapred-site.xml)<=
/b></div><div><b>=C2=A0io.sort.mb =3D 256M (mapred-site.xml)<br clear=3D"al=
l"></b>=C2=A0</div><div>I want to know <b><font class=3D"Apple-style-span" =
size=3D"4">whether these configurations are suitable (should they be config=
ured smaller or bigger ?) for the dataNode machines</font></b>, and <font c=
lass=3D"Apple-style-span" size=3D"4"><b>is there any other configurations w=
hich can take more advantage of the =C2=A0large memory</b>;</font>=C2=A0</d=
iv>
<div><br></div><div>thank you for your reply in advance.</div><div><br></di=
v><div><br></div><div>-- <br>Stay Hungry. Stay Foolish.<br><br>
</div>

--bcaec5215f27d47b8404a256a05a--

--bcaec5215f27d47b8a04a256a05c--