Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 84441F1B4 for ; Wed, 17 Apr 2013 13:22:36 +0000 (UTC) Received: (qmail 27780 invoked by uid 500); 17 Apr 2013 13:22:31 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 27045 invoked by uid 500); 17 Apr 2013 13:22:28 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Delivered-To: moderator for user@hadoop.apache.org Received: (qmail 91014 invoked by uid 99); 17 Apr 2013 13:11:56 -0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of marcosm@buscapecompany.com designates 187.95.198.220 as permitted sender) From: MARCOS MEDRADO RUBINELLI To: "user@hadoop.apache.org" Subject: Re: Adjusting tasktracker heap size? Thread-Topic: Adjusting tasktracker heap size? Thread-Index: AQHOOcnhAzn71OHYf0OCAJAUak5A5g== Date: Wed, 17 Apr 2013 13:11:28 +0000 Message-ID: <516E9F71.7040208@buscapecompany.com> References: <516BE036.5090807@buscapecompany.com> In-Reply-To: Accept-Language: pt-BR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [189.125.42.154] Content-Type: multipart/alternative; boundary="_000_516E9F717040208buscapecompanycom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_516E9F717040208buscapecompanycom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you for the replies. Thankfully, this cluster works with a fairly reg= ular load, so it shouldn't be too hard to fine-tune. Regards, Marcos On 17-04-2013 09:23, Bejoy Ks wrote: Hi Marcos, You need to consider the slots based on the available memory Available Memory =3D Total RAM - (Memory for OS + Memory for Hadoop Daemons= like DN,TT + Memory for other servicess if any running in that node) Now you need to consider the generic MR jobs planned on your cluster. Say i= f your tasks need 1G of JVM to run gracefully, then Possible number of slots =3D Available Memory / JVM size of each task Now divide the slots between mappers and reducers. On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose > wrote: It depends on the type of job that is frequently submitting. RAM size of the machine. Heap size of tasktracker=3D (mapslots+reduceslots)*jvm size We can adjust this according to our requirement to fine tune our cluster. This is my thought. On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI > wrote: Hi, I am currently tuning a cluster, and I haven't found much information on what factors to consider while adjusting the heap size of tasktrackers. Is it a direct multiple of the number of map+reduce slots? Is there anything else I should consider? Thank you, Marcos --_000_516E9F717040208buscapecompanycom_ Content-Type: text/html; charset="iso-8859-1" Content-ID: Content-Transfer-Encoding: quoted-printable

Thank you for the replies. Thankfully, this = cluster works with a fairly regular load, so it shouldn't be too hard to fi= ne-tune.

Regards,
Marcos

On 17-04-2013 09:23, Bejoy Ks wrote:

Hi Marcos,

You need to consider the slots based on the available memory

Available Memory =3D Total RAM - (Memory for OS + Memory for Hadoop Dae= mons like DN,TT + Memory for other servicess if any running in that nod= e)

Now you need to consider the generic MR jobs planned on your cluster. Say i= f your tasks need 1G of JVM to run gracefully, then

Possible number of slots =3D Available Memory / JVM size of each task

Now divide the slots between mappers and reducers.

On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <amalgjos@gmail.com> wrote:

It depends on the type of job that is frequently submittin= g.
RAM size of the machine.

Heap size of tasktracker=3D (mapslots+reduceslots)*jvm size

We can adjust this according to our requirement to fine tune our clust= er.

This is my thought.

On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO = RUBINELLI <marcosm@buscapecompany.com> wrote:

Hi,

I am currently tuning a cluster, and I haven't found much information on what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?

Thank you,
Marcos

--_000_516E9F717040208buscapecompanycom_--