Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4294710CBB for ; Tue, 18 Jun 2013 08:59:38 +0000 (UTC) Received: (qmail 34791 invoked by uid 500); 18 Jun 2013 08:59:32 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 34402 invoked by uid 500); 18 Jun 2013 08:59:24 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 34394 invoked by uid 99); 18 Jun 2013 08:59:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jun 2013 08:59:23 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of samliuhadoop@gmail.com designates 74.125.83.43 as permitted sender) Received: from [74.125.83.43] (HELO mail-ee0-f43.google.com) (74.125.83.43) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jun 2013 08:59:16 +0000 Received: by mail-ee0-f43.google.com with SMTP id l10so2344072eei.16 for ; Tue, 18 Jun 2013 01:58:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hKt1sir0ErRVveJr6KM1818MJXMNz7Pa7qnlMIX0IR8=; b=lxgHUgGoU5YkXEHezUWTK1BPA8j0ZJjvHny3PJJPIom/pvh7mEX1lmLOL/XeQOb1iM BOG31U9mU6FvewK8RV0EWzkrXgsb8yeeyvAK5uSECSRMZ0zLaB+pzGgm2u5Cvt7eniiY syAVWr26K25E2E3bfVkYfRDRNftPiNC/G58HSa1rA2L15yHiDXiFwY1sW3d8Ko8bxPuM DhpndMS1Y3ypRJYJQUNA99BxeExfIZQAop+vHF3uj5s2F6QII8c/exBi9t72R3RlzLQz 78UbI4O6sKPU/idfbPkI0cUMdLJBVB/Xhtt4Pe2OMVvU4G4T4eP6F2I6zbaUqVoP4lQC k4pA== MIME-Version: 1.0 X-Received: by 10.15.21.78 with SMTP id c54mr21287424eeu.14.1371545936433; Tue, 18 Jun 2013 01:58:56 -0700 (PDT) Received: by 10.14.223.137 with HTTP; Tue, 18 Jun 2013 01:58:56 -0700 (PDT) In-Reply-To: References: Date: Tue, 18 Jun 2013 16:58:56 +0800 Message-ID: Subject: Re: Why my tests shows Yarn is worse than MRv1 for terasort? From: sam liu To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=089e0158c9da55be6504df69ec05 X-Virus-Checked: Checked by ClamAV on apache.org --089e0158c9da55be6504df69ec05 Content-Type: text/plain; charset=ISO-8859-1 Hi Harsh, Thanks for your detailed response! Now, the efficiency of my Yarn cluster improved a lot after increasing the reducer number(mapreduce.job.reduces) in mapred-site.xml. But I still have some questions about the way of Yarn to execute MRv1 job: 1.In Hadoop 1.x, a job will be executed by map task and reduce task together, with a typical process(map > shuffle > reduce). In Yarn, as I know, a MRv1 job will be executed only by ApplicationMaster. - Yarn could run multiple kinds of jobs(MR, MPI, ...), but, MRv1 job has special execution process(map > shuffle > reduce) in Hadoop 1.x, and how Yarn execute a MRv1 job? still include some special MR steps in Hadoop 1.x, like map, sort, merge, combine and shuffle? - Do the MRv1 parameters still work for Yarn? Like mapreduce.task.io.sort.mb and mapreduce.map.sort.spill.percent? - What's the general process for ApplicationMaster of Yarn to execute a job? 2. In Hadoop 1.x, we can set the map/reduce slots by setting 'mapred.tasktracker.map.tasks.maximum' and 'mapred.tasktracker.reduce.tasks.maximum' - For Yarn, above tow parameter do not work any more, as yarn uses container instead, right? - For Yarn, we can set the whole physical mem for a NodeManager using ' yarn.nodemanager.resource.memory-mb'. But how to set the default size of physical mem of a container? - How to set the maximum size of physical mem of a container? By the parameter of 'mapred.child.java.opts'? Thanks as always! 2013/6/9 Harsh J > Hi Sam, > > > - How to know the container number? Why you say it will be 22 containers > due to a 22 GB memory? > > The MR2's default configuration requests 1 GB resource each for Map > and Reduce containers. It requests 1.5 GB for the AM container that > runs the job, additionally. This is tunable using the properties > Sandy's mentioned in his post. > > > - My machine has 32 GB memory, how many memory is proper to be assigned > to containers? > > This is a general question. You may use the same process you took to > decide optimal number of slots in MR1 to decide this here. Every > container is a new JVM, and you're limited by the CPUs you have there > (if not the memory). Either increase memory requests from jobs, to > lower # of concurrent containers at a given time (runtime change), or > lower NM's published memory resources to control the same (config > change). > > > - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn', > will other parameters for mapred-site.xml still work in yarn framework? > Like 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent' > > Yes, all of these properties will still work. Old properties specific > to JobTracker or TaskTracker (usually found as a keyword in the config > name) will not apply anymore. > > On Sun, Jun 9, 2013 at 2:21 PM, sam liu wrote: > > Hi Harsh, > > > > According to above suggestions, I removed the duplication of setting, and > > reduce the value of 'yarn.nodemanager.resource.cpu-cores', > > 'yarn.nodemanager.vcores-pcores-ratio' and > > 'yarn.nodemanager.resource.memory-mb' to 16, 8 and 12000. Ant then, the > > efficiency improved about 18%. I have questions: > > > > - How to know the container number? Why you say it will be 22 containers > due > > to a 22 GB memory? > > - My machine has 32 GB memory, how many memory is proper to be assigned > to > > containers? > > - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn', > will > > other parameters for mapred-site.xml still work in yarn framework? Like > > 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent' > > > > Thanks! > > > > > > > > 2013/6/8 Harsh J > >> > >> Hey Sam, > >> > >> Did you get a chance to retry with Sandy's suggestions? The config > >> appears to be asking NMs to use roughly 22 total containers (as > >> opposed to 12 total tasks in MR1 config) due to a 22 GB memory > >> resource. This could impact much, given the CPU is still the same for > >> both test runs. > >> > >> On Fri, Jun 7, 2013 at 12:23 PM, Sandy Ryza > >> wrote: > >> > Hey Sam, > >> > > >> > Thanks for sharing your results. I'm definitely curious about what's > >> > causing the difference. > >> > > >> > A couple observations: > >> > It looks like you've got yarn.nodemanager.resource.memory-mb in there > >> > twice > >> > with two different values. > >> > > >> > Your max JVM memory of 1000 MB is (dangerously?) close to the default > >> > mapreduce.map/reduce.memory.mb of 1024 MB. Are any of your tasks > getting > >> > killed for running over resource limits? > >> > > >> > -Sandy > >> > > >> > > >> > On Thu, Jun 6, 2013 at 10:21 PM, sam liu > wrote: > >> >> > >> >> The terasort execution log shows that reduce spent about 5.5 mins > from > >> >> 33% > >> >> to 35% as below. > >> >> 13/06/10 08:02:22 INFO mapreduce.Job: map 100% reduce 31% > >> >> 13/06/10 08:02:25 INFO mapreduce.Job: map 100% reduce 32% > >> >> 13/06/10 08:02:46 INFO mapreduce.Job: map 100% reduce 33% > >> >> 13/06/10 08:08:16 INFO mapreduce.Job: map 100% reduce 35% > >> >> 13/06/10 08:08:19 INFO mapreduce.Job: map 100% reduce 40% > >> >> 13/06/10 08:08:22 INFO mapreduce.Job: map 100% reduce 43% > >> >> > >> >> Any way, below are my configurations for your reference. Thanks! > >> >> (A) core-site.xml > >> >> only define 'fs.default.name' and 'hadoop.tmp.dir' > >> >> > >> >> (B) hdfs-site.xml > >> >> > >> >> dfs.replication > >> >> 1 > >> >> > >> >> > >> >> > >> >> dfs.name.dir > >> >> /opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_name_dir > >> >> > >> >> > >> >> > >> >> dfs.data.dir > >> >> /opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_data_dir > >> >> > >> >> > >> >> > >> >> dfs.block.size > >> >> 134217728 > >> >> > >> >> > >> >> > >> >> dfs.namenode.handler.count > >> >> 64 > >> >> > >> >> > >> >> > >> >> dfs.datanode.handler.count > >> >> 10 > >> >> > >> >> > >> >> (C) mapred-site.xml > >> >> > >> >> mapreduce.cluster.temp.dir > >> >> /opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduce_temp > >> >> No description > >> >> true > >> >> > >> >> > >> >> > >> >> mapreduce.cluster.local.dir > >> >> > >> >> > /opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduce_local_dir > >> >> No description > >> >> true > >> >> > >> >> > >> >> > >> >> mapreduce.child.java.opts > >> >> -Xmx1000m > >> >> > >> >> > >> >> > >> >> mapreduce.framework.name > >> >> yarn > >> >> > >> >> > >> >> > >> >> mapreduce.tasktracker.map.tasks.maximum > >> >> 8 > >> >> > >> >> > >> >> > >> >> mapreduce.tasktracker.reduce.tasks.maximum > >> >> 4 > >> >> > >> >> > >> >> > >> >> > >> >> mapreduce.tasktracker.outofband.heartbeat > >> >> true > >> >> > >> >> > >> >> (D) yarn-site.xml > >> >> > >> >> yarn.resourcemanager.resource-tracker.address > >> >> node1:18025 > >> >> host is the hostname of the resource manager and > >> >> port is the port on which the NodeManagers contact the Resource > >> >> Manager. > >> >> > >> >> > >> >> > >> >> > >> >> The address of the RM web application. > >> >> yarn.resourcemanager.webapp.address > >> >> node1:18088 > >> >> > >> >> > >> >> > >> >> > >> >> yarn.resourcemanager.scheduler.address > >> >> node1:18030 > >> >> host is the hostname of the resourcemanager and port > >> >> is > >> >> the port > >> >> on which the Applications in the cluster talk to the Resource > >> >> Manager. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> yarn.resourcemanager.address > >> >> node1:18040 > >> >> the host is the hostname of the ResourceManager and > >> >> the > >> >> port is the port on > >> >> which the clients can talk to the Resource Manager. > > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.local-dirs > >> >> > >> >> /opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_local_dir > >> >> the local directories used by the > >> >> nodemanager > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.address > >> >> 0.0.0.0:18050 > >> >> the nodemanagers bind to this port > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.resource.memory-mb > >> >> 10240 > >> >> the amount of memory on the NodeManager in > >> >> GB > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.remote-app-log-dir > >> >> > /opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_app-logs > >> >> directory on hdfs where the application logs are > moved > >> >> to > >> >> > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.log-dirs > >> >> /opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_log > >> >> the directories used by Nodemanagers as log > >> >> directories > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.aux-services > >> >> mapreduce.shuffle > >> >> shuffle service that needs to be set for Map Reduce > to > >> >> run > >> >> > >> >> > >> >> > >> >> yarn.resourcemanager.client.thread-count > >> >> 64 > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.resource.cpu-cores > >> >> 24 > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.vcores-pcores-ratio > >> >> 3 > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.resource.memory-mb > >> >> 22000 > >> >> > >> >> > >> >> > >> >> yarn.nodemanager.vmem-pmem-ratio > >> >> 2.1 > >> >> > >> >> > >> >> > >> >> > >> >> 2013/6/7 Harsh J > >> >>> > >> >>> Not tuning configurations at all is wrong. YARN uses memory resource > >> >>> based scheduling and hence MR2 would be requesting 1 GB minimum by > >> >>> default, causing, on base configs, to max out at 8 (due to 8 GB NM > >> >>> memory resource config) total containers. Do share your configs as > at > >> >>> this point none of us can tell what it is. > >> >>> > >> >>> Obviously, it isn't our goal to make MR2 slower for users and to not > >> >>> care about such things :) > >> >>> > >> >>> On Fri, Jun 7, 2013 at 8:45 AM, sam liu > >> >>> wrote: > >> >>> > At the begining, I just want to do a fast comparision of MRv1 and > >> >>> > Yarn. > >> >>> > But > >> >>> > they have many differences, and to be fair for comparison I did > not > >> >>> > tune > >> >>> > their configurations at all. So I got above test results. After > >> >>> > analyzing > >> >>> > the test result, no doubt, I will configure them and do comparison > >> >>> > again. > >> >>> > > >> >>> > Do you have any idea on current test result? I think, to compare > >> >>> > with > >> >>> > MRv1, > >> >>> > Yarn is better on Map phase(teragen test), but worse on Reduce > >> >>> > phase(terasort test). > >> >>> > And any detailed suggestions/comments/materials on Yarn > performance > >> >>> > tunning? > >> >>> > > >> >>> > Thanks! > >> >>> > > >> >>> > > >> >>> > 2013/6/7 Marcos Luis Ortiz Valmaseda > >> >>> >> > >> >>> >> Why not to tune the configurations? > >> >>> >> Both frameworks have many areas to tune: > >> >>> >> - Combiners, Shuffle optimization, Block size, etc > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> 2013/6/6 sam liu > >> >>> >>> > >> >>> >>> Hi Experts, > >> >>> >>> > >> >>> >>> We are thinking about whether to use Yarn or not in the near > >> >>> >>> future, > >> >>> >>> and > >> >>> >>> I ran teragen/terasort on Yarn and MRv1 for comprison. > >> >>> >>> > >> >>> >>> My env is three nodes cluster, and each node has similar > hardware: > >> >>> >>> 2 > >> >>> >>> cpu(4 core), 32 mem. Both Yarn and MRv1 cluster are set on the > >> >>> >>> same > >> >>> >>> env. To > >> >>> >>> be fair, I did not make any performance tuning on their > >> >>> >>> configurations, but > >> >>> >>> use the default configuration values. > >> >>> >>> > >> >>> >>> Before testing, I think Yarn will be much better than MRv1, if > >> >>> >>> they > >> >>> >>> all > >> >>> >>> use default configuration, because Yarn is a better framework > than > >> >>> >>> MRv1. > >> >>> >>> However, the test result shows some differences: > >> >>> >>> > >> >>> >>> MRv1: Hadoop-1.1.1 > >> >>> >>> Yarn: Hadoop-2.0.4 > >> >>> >>> > >> >>> >>> (A) Teragen: generate 10 GB data: > >> >>> >>> - MRv1: 193 sec > >> >>> >>> - Yarn: 69 sec > >> >>> >>> Yarn is 2.8 times better than MRv1 > >> >>> >>> > >> >>> >>> (B) Terasort: sort 10 GB data: > >> >>> >>> - MRv1: 451 sec > >> >>> >>> - Yarn: 1136 sec > >> >>> >>> Yarn is 2.5 times worse than MRv1 > >> >>> >>> > >> >>> >>> After a fast analysis, I think the direct cause might be that > Yarn > >> >>> >>> is > >> >>> >>> much faster than MRv1 on Map phase, but much worse on Reduce > >> >>> >>> phase. > >> >>> >>> > >> >>> >>> Here I have two questions: > >> >>> >>> - Why my tests shows Yarn is worse than MRv1 for terasort? > >> >>> >>> - What's the stratage for tuning Yarn performance? Is any > >> >>> >>> materials? > >> >>> >>> > >> >>> >>> Thanks! > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> -- > >> >>> >> Marcos Ortiz Valmaseda > >> >>> >> Product Manager at PDVSA > >> >>> >> http://about.me/marcosortiz > >> >>> >> > >> >>> > > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> Harsh J > >> >> > >> >> > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J > --089e0158c9da55be6504df69ec05 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Harsh,

Thanks for your deta= iled response! Now, the efficiency of my Yarn cluster improved a lot after = increasing the reducer number(mapreduce.job.reduces) in mapred-site.xml. Bu= t I still have some questions about the way of Yarn to execute MRv1 job:
1.In Hadoop 1.x, a job will be executed by map task and reduce task tog= ether, with a typical process(map > shuffle > reduce). In Yarn, as I = know, a MRv1 job will be executed only by ApplicationMaster.
- Yarn coul= d run multiple kinds of jobs(MR, MPI, ...), but, MRv1 job has special execu= tion process(map > shuffle > reduce) in Hadoop 1.x, and how Yarn exec= ute a MRv1 job? still include some special MR steps in Hadoop 1.x, like map= , sort, merge, combine and shuffle?
- Do the MRv1 parameters still work for Yarn? Like mapreduce.task.io.sort.m= b and mapreduce.map.sort.spill.percent?
- What's the general process= for ApplicationMaster of Yarn to execute a job?

2. In Hadoop 1.x, = we can set the map/reduce slots by setting 'mapred.tasktracker.map.task= s.maximum' and 'mapred.tasktracker.reduce.tasks.maximum'
- For Yarn, above tow parameter do not work any more, as yarn uses co= ntainer instead, right?
- For Yarn, we can set the whole physical = mem for a NodeManager using 'yarn.nodemanager.resource.memory-mb'. But how = to set the default size of physical mem of a container?
- How to set the maximum size of physical mem of a container? By = the parameter of 'mapred.child.java.opts'?

Thanks as always!

2013/6/9 Ha= rsh J <harsh@cloudera.com>
Hi Sam,

> - How to know the container number? Why you say it will be 22 containe= rs due to a 22 GB memory?

The MR2's default configuration requests 1 GB resource each for M= ap
and Reduce containers. It requests 1.5 GB for the AM container that
runs the job, additionally. This is tunable using the properties
Sandy's mentioned in his post.

> - My machine has 32 GB memory, how many memory is proper to be assigne= d to containers?

This is a general question. You may use the same process you took to<= br> decide optimal number of slots in MR1 to decide this here. Every
container is a new JVM, and you're limited by the CPUs you have there (if not the memory). Either increase memory requests from jobs, to
lower # of concurrent containers at a given time (runtime change), or
lower NM's published memory resources to control the same (config
change).

> - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'ya= rn', will other parameters for mapred-site.xml still work in yarn frame= work? Like 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.= spill.percent'

Yes, all of these properties will still work. Old properties specific=
to JobTracker or TaskTracker (usually found as a keyword in the config
name) will not apply anymore.

On Sun, Jun 9, 2013 at 2:21 PM, sam liu <samliuhadoop@gmail.com> wrote:
> Hi Harsh,
>
> According to above suggestions, I removed the duplication of setting, = and
> reduce the value of 'yarn.nodemanager.resource.cpu-cores',
> 'yarn.nodemanager.vcores-pcores-ratio' and
> 'yarn.nodemanager.resource.memory-mb' to 16, 8 and 12000. Ant = then, the
> efficiency improved about 18%. =A0I have questions:
>
> - How to know the container number? Why you say it will be 22 containe= rs due
> to a 22 GB memory?
> - My machine has 32 GB memory, how many memory is proper to be assigne= d to
> containers?
> - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'ya= rn', will
> other parameters for mapred-site.xml still work in yarn framework? Lik= e
> 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.= percent'
>
> Thanks!
>
>
>
> 2013/6/8 Harsh J <harsh@cloud= era.com>
>>
>> Hey Sam,
>>
>> Did you get a chance to retry with Sandy's suggestions? The co= nfig
>> appears to be asking NMs to use roughly 22 total containers (as >> opposed to 12 total tasks in MR1 config) due to a 22 GB memory
>> resource. This could impact much, given the CPU is still the same = for
>> both test runs.
>>
>> On Fri, Jun 7, 2013 at 12:23 PM, Sandy Ryza <sandy.ryza@cloudera.com>
>> wrote:
>> > Hey Sam,
>> >
>> > Thanks for sharing your results. =A0I'm definitely curiou= s about what's
>> > causing the difference.
>> >
>> > A couple observations:
>> > It looks like you've got yarn.nodemanager.resource.memory= -mb in there
>> > twice
>> > with two different values.
>> >
>> > Your max JVM memory of 1000 MB is (dangerously?) close to the= default
>> > mapreduce.map/reduce.memory.mb of 1024 MB. Are any of your ta= sks getting
>> > killed for running over resource limits?
>> >
>> > -Sandy
>> >
>> >
>> > On Thu, Jun 6, 2013 at 10:21 PM, sam liu <samliuhadoop@gmail.com> wrote:
>> >>
>> >> The terasort execution log shows that reduce spent about = 5.5 mins from
>> >> 33%
>> >> to 35% as below.
>> >> 13/06/10 08:02:22 INFO mapreduce.Job: =A0map 100% reduce = 31%
>> >> 13/06/10 08:02:25 INFO mapreduce.Job: =A0map 100% reduce = 32%
>> >> 13/06/10 08:02:46 INFO mapreduce.Job: =A0map 100% reduce = 33%
>> >> 13/06/10 08:08:16 INFO mapreduce.Job: =A0map 100% reduce = 35%
>> >> 13/06/10 08:08:19 INFO mapreduce.Job: =A0map 100% reduce = 40%
>> >> 13/06/10 08:08:22 INFO mapreduce.Job: =A0map 100% reduce = 43%
>> >>
>> >> Any way, below are my configurations for your reference. = Thanks!
>> >> (A) core-site.xml
>> >> only define 'fs.default.name' and 'hadoop.tmp.dir'
>> >>
>> >> (B) hdfs-site.xml
>> >> =A0 <property>
>> >> =A0 =A0 <name>dfs.replication</name>
>> >> =A0 =A0 <value>1</value>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>dfs.name.dir</name>
>> >> =A0 =A0 <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/= dfs_name_dir</value>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>dfs.data.dir</name>
>> >> =A0 =A0 <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/= dfs_data_dir</value>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>dfs.block.size</name>
>> >> =A0 =A0 <value>134217728</value><!-- 128MB= -->
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>dfs.namenode.handler.count</name&g= t;
>> >> =A0 =A0 <value>64</value>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>dfs.datanode.handler.count</name&g= t;
>> >> =A0 =A0 <value>10</value>
>> >> =A0 </property>
>> >>
>> >> (C) mapred-site.xml
>> >> =A0 <property>
>> >> =A0 =A0 <name>mapreduce.cluster.temp.dir</name&g= t;
>> >> =A0 =A0 <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/= mapreduce_temp</value>
>> >> =A0 =A0 <description>No description</description= >
>> >> =A0 =A0 <final>true</final>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>mapreduce.cluster.local.dir</name&= gt;
>> >>
>> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduc= e_local_dir</value>
>> >> =A0 =A0 <description>No description</description= >
>> >> =A0 =A0 <final>true</final>
>> >> =A0 </property>
>> >>
>> >> <property>
>> >> =A0 <name>mapreduce.child.java.opts</name> >> >> =A0 <value>-Xmx1000m</value>
>> >> </property>
>> >>
>> >> <property>
>> >> =A0 =A0 <name>mapreduce.framework.name</name>
>> >> =A0 =A0 <value>yarn</value>
>> >> =A0 =A0</property>
>> >>
>> >> =A0<property>
>> >> =A0 =A0 <name>mapreduce.tasktracker.map.tasks.maxim= um</name>
>> >> =A0 =A0 <value>8</value>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>mapreduce.tasktracker.reduce.tasks.ma= ximum</name>
>> >> =A0 =A0 <value>4</value>
>> >> =A0 </property>
>> >>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>mapreduce.tasktracker.outofband.heart= beat</name>
>> >> =A0 =A0 <value>true</value>
>> >> =A0 </property>
>> >>
>> >> (D) yarn-site.xml
>> >> =A0<property>
>> >> =A0 =A0 <name>yarn.resourcemanager.resource-tracker= .address</name>
>> >> =A0 =A0 <value>node1:18025</value>
>> >> =A0 =A0 <description>host is the hostname of the re= source manager and
>> >> =A0 =A0 port is the port on which the NodeManagers contac= t the Resource
>> >> Manager.
>> >> =A0 =A0 </description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <description>The address of the RM web appl= ication.</description>
>> >> =A0 =A0 <name>yarn.resourcemanager.webapp.address&l= t;/name>
>> >> =A0 =A0 <value>node1:18088</value>
>> >> =A0 </property>
>> >>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.resourcemanager.scheduler.addres= s</name>
>> >> =A0 =A0 <value>node1:18030</value>
>> >> =A0 =A0 <description>host is the hostname of the re= sourcemanager and port
>> >> is
>> >> the port
>> >> =A0 =A0 on which the Applications in the cluster talk to = the Resource
>> >> Manager.
>> >> =A0 =A0 </description>
>> >> =A0 </property>
>> >>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.resourcemanager.address</name= >
>> >> =A0 =A0 <value>node1:18040</value>
>> >> =A0 =A0 <description>the host is the hostname of th= e ResourceManager and
>> >> the
>> >> port is the port on
>> >> =A0 =A0 which the clients can talk to the Resource Manage= r. </description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.nodemanager.local-dirs</name&= gt;
>> >>
>> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_= local_dir</value>
>> >> =A0 =A0 <description>the local directories used by = the
>> >> nodemanager</description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.nodemanager.address</name>=
>> >> =A0 =A0 <value>0.0.0.0:18050</value>
>> >> =A0 =A0 <description>the nodemanagers bind to this = port</description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.nodemanager.resource.memory-mb&l= t;/name>
>> >> =A0 =A0 <value>10240</value>
>> >> =A0 =A0 <description>the amount of memory on the No= deManager in
>> >> GB</description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.nodemanager.remote-app-log-dir&l= t;/name>
>> >> =A0 =A0 <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/= yarn_nm_app-logs</value>
>> >> =A0 =A0 <description>directory on hdfs where the ap= plication logs are moved
>> >> to
>> >> </description>
>> >> =A0 </property>
>> >>
>> >> =A0 =A0<property>
>> >> =A0 =A0 <name>yarn.nodemanager.log-dirs</name>= ;
>> >> =A0 =A0 <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/= yarn_nm_log</value>
>> >> =A0 =A0 <description>the directories used by Nodema= nagers as log
>> >> directories</description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.nodemanager.aux-services</nam= e>
>> >> =A0 =A0 <value>mapreduce.shuffle</value>
>> >> =A0 =A0 <description>shuffle service that needs to = be set for Map Reduce to
>> >> run </description>
>> >> =A0 </property>
>> >>
>> >> =A0 <property>
>> >> =A0 =A0 <name>yarn.resourcemanager.client.thread-co= unt</name>
>> >> =A0 =A0 <value>64</value>
>> >> =A0 </property>
>> >>
>> >> =A0<property>
>> >> =A0 =A0 <name>yarn.nodemanager.resource.cpu-cores&l= t;/name>
>> >> =A0 =A0 <value>24</value>
>> >> =A0 </property>
>> >>
>> >> <property>
>> >> =A0 =A0 <name>yarn.nodemanager.vcores-pcores-ratio&= lt;/name>
>> >> =A0 =A0 <value>3</value>
>> >> =A0 </property>
>> >>
>> >> =A0<property>
>> >> =A0 =A0 <name>yarn.nodemanager.resource.memory-mb&l= t;/name>
>> >> =A0 =A0 <value>22000</value>
>> >> =A0 </property>
>> >>
>> >> =A0<property>
>> >> =A0 =A0 <name>yarn.nodemanager.vmem-pmem-ratio</= name>
>> >> =A0 =A0 <value>2.1</value>
>> >> =A0 </property>
>> >>
>> >>
>> >>
>> >> 2013/6/7 Harsh J <harsh@cloudera.com>
>> >>>
>> >>> Not tuning configurations at all is wrong. YARN uses = memory resource
>> >>> based scheduling and hence MR2 would be requesting 1 = GB minimum by
>> >>> default, causing, on base configs, to max out at 8 (d= ue to 8 GB NM
>> >>> memory resource config) total containers. Do share yo= ur configs as at
>> >>> this point none of us can tell what it is.
>> >>>
>> >>> Obviously, it isn't our goal to make MR2 slower f= or users and to not
>> >>> care about such things :)
>> >>>
>> >>> On Fri, Jun 7, 2013 at 8:45 AM, sam liu <samliuhadoop@gmail.com>
>> >>> wrote:
>> >>> > At the begining, I just want to do a fast compar= ision of MRv1 and
>> >>> > Yarn.
>> >>> > But
>> >>> > they have many differences, and to be fair for c= omparison I did not
>> >>> > tune
>> >>> > their configurations at all. =A0So I got above t= est results. After
>> >>> > analyzing
>> >>> > the test result, no doubt, I will configure them= and do comparison
>> >>> > again.
>> >>> >
>> >>> > Do you have any idea on current test result? I t= hink, to compare
>> >>> > with
>> >>> > MRv1,
>> >>> > Yarn is better on Map phase(teragen test), but w= orse on Reduce
>> >>> > phase(terasort test).
>> >>> > And any detailed suggestions/comments/materials = on Yarn performance
>> >>> > tunning?
>> >>> >
>> >>> > Thanks!
>> >>> >
>> >>> >
>> >>> > 2013/6/7 Marcos Luis Ortiz Valmaseda <marcosluis2186@gmail.com>
>> >>> >>
>> >>> >> Why not to tune the configurations?
>> >>> >> Both frameworks have many areas to tune:
>> >>> >> - Combiners, Shuffle optimization, Block siz= e, etc
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> 2013/6/6 sam liu <samliuhadoop@gmail.com>
>> >>> >>>
>> >>> >>> Hi Experts,
>> >>> >>>
>> >>> >>> We are thinking about whether to use Yar= n or not in the near
>> >>> >>> future,
>> >>> >>> and
>> >>> >>> I ran teragen/terasort on Yarn and MRv1 = for comprison.
>> >>> >>>
>> >>> >>> My env is three nodes cluster, and each = node has similar hardware:
>> >>> >>> 2
>> >>> >>> cpu(4 core), 32 mem. Both Yarn and MRv1 = cluster are set on the
>> >>> >>> same
>> >>> >>> env. To
>> >>> >>> be fair, I did not make any performance = tuning on their
>> >>> >>> configurations, but
>> >>> >>> use the default configuration values. >> >>> >>>
>> >>> >>> Before testing, I think Yarn will be muc= h better than MRv1, if
>> >>> >>> they
>> >>> >>> all
>> >>> >>> use default configuration, because Yarn = is a better framework than
>> >>> >>> MRv1.
>> >>> >>> However, the test result shows some diff= erences:
>> >>> >>>
>> >>> >>> MRv1: Hadoop-1.1.1
>> >>> >>> Yarn: Hadoop-2.0.4
>> >>> >>>
>> >>> >>> (A) Teragen: generate 10 GB data:
>> >>> >>> - MRv1: 193 sec
>> >>> >>> - Yarn: 69 sec
>> >>> >>> Yarn is 2.8 times better than MRv1
>> >>> >>>
>> >>> >>> (B) Terasort: sort 10 GB data:
>> >>> >>> - MRv1: 451 sec
>> >>> >>> - Yarn: 1136 sec
>> >>> >>> Yarn is 2.5 times worse than MRv1
>> >>> >>>
>> >>> >>> After a fast analysis, I think the direc= t cause might be that Yarn
>> >>> >>> is
>> >>> >>> much faster than MRv1 on Map phase, but = much worse on Reduce
>> >>> >>> phase.
>> >>> >>>
>> >>> >>> Here I have two questions:
>> >>> >>> - Why my tests shows Yarn is worse than = MRv1 for terasort?
>> >>> >>> - What's the stratage for tuning Yar= n performance? Is any
>> >>> >>> materials?
>> >>> >>>
>> >>> >>> Thanks!
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Marcos Ortiz Valmaseda
>> >>> >> Product Manager at PDVSA
>> >>> >> http://about.me/marcosortiz
>> >>> >>
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



--
Harsh J

--089e0158c9da55be6504df69ec05--