Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1949AD167 for ; Thu, 16 Aug 2012 18:43:02 +0000 (UTC) Received: (qmail 29031 invoked by uid 500); 16 Aug 2012 18:42:57 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 28915 invoked by uid 500); 16 Aug 2012 18:42:57 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 28908 invoked by uid 99); 16 Aug 2012 18:42:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Aug 2012 18:42:57 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.161.176] (HELO mail-gg0-f176.google.com) (209.85.161.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Aug 2012 18:42:48 +0000 Received: by ggke5 with SMTP id e5so4114891ggk.35 for ; Thu, 16 Aug 2012 11:42:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:mime-version:content-type:subject:date:in-reply-to:to :references:message-id:x-mailer:x-gm-message-state; bh=hm8ahrgYYXKeyCLcJ6wPt7INyOltmkad/EdVOhbRLQU=; b=i8JDvcoNOqyr4r9MByf3EB+mYQC1SSvyavHY0CvwqmhKCJInoP17EklnttmMszxtxU krSVKx76BTHZOqJ7Z3sFOSTXeeBfPAJWP0P/DAITMoJn9yc27zIDF7Db3nBhxvin55XR cNVxc63MWFt2bjmpmo84gjECSVqPFLil2AYb2j6hcc1PHDCxAtZwRum2eaILtmHA6cec mqMeCRHKbZmTL1y/OVlOSOM5N8e2iv+9FsCgJg5+MxRKftbtS1R6kAWIBVNVplvI6/YW IJg/D4Y5DxwPN7rJ125STbgfy9HLUUJG3B35wqrQAyVhN5u+MDnLdSXrteSwvC5Ve47Y Z67Q== Received: by 10.66.73.98 with SMTP id k2mr3853026pav.73.1345142545728; Thu, 16 Aug 2012 11:42:25 -0700 (PDT) Received: from [10.10.11.145] (host1.hortonworks.com. [70.35.59.2]) by mx.google.com with ESMTPS id qb6sm2233572pbb.18.2012.08.16.11.42.22 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 16 Aug 2012 11:42:23 -0700 (PDT) From: Arun C Murthy Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-466--384362086 Subject: Re: Relevance of mapreduce.* configuration properties for MR V2 Date: Thu, 16 Aug 2012 11:42:21 -0700 In-Reply-To: <502D2FC4.6020902@gmail.com> To: user@hadoop.apache.org References: <502D1FC9.9030704@gmail.com> <502D2FC4.6020902@gmail.com> Message-Id: X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnvPEITnILW6QGZnVicoZUR2Im2+E2cZxSQ5y+um6/MQMul1yWQMu+cCVx2hfgUJwBO8MUk --Apple-Mail-466--384362086 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii h= ttp://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-s= ite/ClusterSetup.html On Aug 16, 2012, at 10:37 AM, mg wrote: > Hi, >=20 > thanks. >=20 > But there are many more mapreduce.* properties. ... >=20 > Does anyone have more information on those with respect to whether = they are relevant to YARN, and if so, what they exactly mean in the YARN = context? >=20 > I am trying to optimize settings in order to use as much RAM and CPU = as possible. The cluster has 8 node managers each with 96GB RAM and 16 = CPUs (cpuinfo: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz) >=20 > Cheers, > Martin >=20 > On 16.08.2012 18:40, anil gupta wrote: >> Hi, >>=20 >> AFAIK, these properties are being ignored by YARN: >> - mapreduce.tasktracker.map.tasks.maximum, >> - mapreduce.tasktracker.reduce.tasks.maximum >>=20 >> Thanks, >> Anil Gupta >>=20 >>=20 >> On Thu, Aug 16, 2012 at 9:28 AM, mg > > wrote: >>=20 >> Hi, >>=20 >> I am currently trying to tune a CDH 4.0.1 (i~ hadoop 2.0.0-alpha) >> cluster running HDFS, YARN, and HBase managed by Cloudera Manager >> 4.0.3 (Free Edition). >>=20 >>=20 >> In CM, there are a number of options for setting mapreduce.* >> configuration properties on the YARN client page. >>=20 >> Some of the properties and their explanations in the GUI still = refer >> to JobTracker and TaskTracker, e.g., >>=20 >> - mapreduce.jobtracker.handler.__count, >> - mapreduce.tasktracker.map.__tasks.maximum, >> - mapreduce.tasktracker.reduce.__tasks.maximum >>=20 >> I wonder whether these and a number of other mapreduce.* (e.g., >> mapreduce.job.reduces) properties are observed by the MR2 >> ApplicationMaster (and how), or not. >>=20 >>=20 >> Can anyone clarify or point to respective documentation? >>=20 >> Thanks, >> Martin >>=20 >>=20 >>=20 >>=20 >> -- >> Thanks & Regards, >> Anil Gupta -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ --Apple-Mail-466--384362086 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii http://hadoop.apache.org/common/docs/r2.0.= 0-alpha/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html

On Aug 16, 2012, at 10:37 AM, mg wrote:

Hi,

thanks.

But there are many more = mapreduce.* properties. ...

Does anyone have more information on = those with respect to whether they are relevant to YARN, and if so, what = they exactly mean in the YARN context?

I am trying to optimize = settings in order to use as much RAM and CPU as possible. The cluster = has 8 node managers each with 96GB RAM and 16 CPUs (cpuinfo: Intel(R) = Xeon(R) CPU E5620 @ 2.40GHz)

Cheers,
Martin

On = 16.08.2012 18:40, anil gupta wrote:
Hi,

AFAIK, these = properties are being ignored by YARN:
- = mapreduce.tasktracker.map.tasks.maximum,
- = mapreduce.tasktracker.reduce.tasks.maximum

Thanks,
Anil = Gupta


On Thu, Aug 16, = 2012 at 9:28 AM, mg = <userformailinglists@gmail.com
<mailto:userformailinglists@gmail.com>> = wrote:

=    Hi,

=    I am currently trying to tune a CDH 4.0.1 (i~ hadoop = 2.0.0-alpha)
=    cluster running HDFS, YARN, and HBase managed by = Cloudera Manager
=    4.0.3 (Free Edition).


=    In CM, there are a number of options for setting = mapreduce.*
=    configuration properties on the YARN client = page.

=    Some of the properties and their explanations in the = GUI still refer
=    to JobTracker and TaskTracker, = e.g.,

=    - = mapreduce.jobtracker.handler.__count,
   - = mapreduce.tasktracker.map.__tasks.maximum,
   - = mapreduce.tasktracker.reduce.__tasks.maximum

=    I wonder whether these and a number of other = mapreduce.* (e.g.,
=    mapreduce.job.reduces) properties are observed by the = MR2
=    ApplicationMaster (and how), or = not.


=    Can anyone clarify or point to respective = documentation?

=    Thanks,
=    Martin




--
Thanks & = Regards,
Anil = Gupta

--
Arun C. Murthy
Hortonworks = Inc.
http://hortonworks.com/


= --Apple-Mail-466--384362086--