Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E91171098E for ; Sun, 22 Dec 2013 13:49:16 +0000 (UTC) Received: (qmail 55689 invoked by uid 500); 22 Dec 2013 13:40:37 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 55326 invoked by uid 500); 22 Dec 2013 13:39:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 55257 invoked by uid 99); 22 Dec 2013 13:39:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Dec 2013 13:39:38 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nitinpawar432@gmail.com designates 209.85.216.179 as permitted sender) Received: from [209.85.216.179] (HELO mail-qc0-f179.google.com) (209.85.216.179) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Dec 2013 13:39:33 +0000 Received: by mail-qc0-f179.google.com with SMTP id i8so3938902qcq.38 for ; Sun, 22 Dec 2013 05:39:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=f2gYD4lEwwSBN1V5fTu1kpMsEkvxDktPuIz2wcv+8zU=; b=IFEzg97KgDSclT39c+Wu/bZ3RqxxWlnm4Cj6Hzqh5+ntfWgNP5Z3bzak/sXqj1F+UI 3m41pxgeenwVnnmbzQrHeIkD4j183JSjmo32NKeKwWnnNmaoHQbDZhuqup1nZxdWIRoP fji6rwQDK1SyJ1hCvXqgutlM6LuCCo29yKRu8sZ2j5jFkEG0r6I3hkBpQQYswGRm9Vx4 ERRhRgsjJlyilVhGjEr8qA14CfMTB0f8lk0LceTabvuFsHeTSwElPrqVamldIJ0o4F/H hWo9zzk/S+77pYgABmDk43vLtFQiGuCxubB8Hoz91WvcQnxCQkOskzLGNNF+G8e7icV7 NULg== MIME-Version: 1.0 X-Received: by 10.224.67.200 with SMTP id s8mr32914008qai.75.1387719552685; Sun, 22 Dec 2013 05:39:12 -0800 (PST) Received: by 10.224.194.5 with HTTP; Sun, 22 Dec 2013 05:39:12 -0800 (PST) In-Reply-To: References: Date: Sun, 22 Dec 2013 19:09:12 +0530 Message-ID: Subject: Re: How to set the handler count related parameters? From: Nitin Pawar To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c3218afc5d1c04ee1fa2ba X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3218afc5d1c04ee1fa2ba Content-Type: text/plain; charset=ISO-8859-1 In general when you have like large hardware machine like the one you have got you can set dfs.namenode.handler.count = 64 (you can increase this in proportion to your RAM) dfs.datanode.handler.count is defaulted to 3 but you can raise it to around 6 to 10. In some blogs I have read that increasing this will increase memory consumption but no performance gains mapreduce.jobtracker.handler.count this is basically you mean the number of server threads you want to run for jobtracker. From the default hadoop site xml it is recommended that you keep this size to around 4% of number of tasktrackers. mapred.child.java.opts i am not sure about this. by default its 200M but this is setting which is set that the child processes of tasktrackers will start with this option if you do not overwrite them from client side. I may be wrong in this. On Sun, Dec 22, 2013 at 6:50 PM, sam liu wrote: > Hi, > > We have 20 nodes cluster(1 namenode, 1 jobtracker, 18 datanodes). Each > node has 20 cpu cores and 64 GB memory. > > How to set the values for following parameters? > > - > > *dfs.namenode.handler.count - dfs.datanode.handler.count- * > *mapreduce.jobtracker.handler.count* > - *mapred.child.java.opts* > > Thanks very much! > -- Nitin Pawar --001a11c3218afc5d1c04ee1fa2ba Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
In general when you have like large hardware machine like = the one you have got=A0

you can set=A0
dfs.na= menode.handler.count =3D 64 (you can increase this in proportion to your RA= M)

dfs.datanode.handler.count is defaulted = to 3 but you can raise it to around 6 to 10. In some blogs I have read that= increasing this will increase memory consumption but no performance gains<= /span>

mapreduce.jobtracker.handler.coun= t this is basically you mean the number of server threads you want to run f= or jobtracker. From the default hadoop site xml it is recommended that you = keep this size to around 4% of number of tasktrackers. =A0

mapred.child.java.opts i am not sure abo= ut this. by default its 200M but this is setting which is set that the chil= d processes of tasktrackers will start with this option if you do not overw= rite them from client side. I may be wrong in this.=A0


<= strong style=3D"font-family:arial,sans-serif;font-size:12.727272033691406px= ">


On Sun, Dec 22, 2013 at 6:50 PM, sam liu <samliuhadoop@gmail.= com> wrote:
Hi,

= We have 20 nodes cluster(1 namenode, 1 jobtracker, 18 datanodes).=A0 Each n= ode has 20 cpu cores and 64 GB memory.

How to set the values for following parameters?

- dfs.na= menode.handler.count
- dfs.datanode.handler.count
- mapreduce.jobtracker.han= dler.count
- mapred.child.java.opts

Thanks very much!



--
Nitin Pawar<= br>
--001a11c3218afc5d1c04ee1fa2ba--