hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sam liu <samliuhad...@gmail.com>
Subject Re: Why my tests shows Yarn is worse than MRv1 for terasort?
Date Tue, 18 Jun 2013 08:58:56 GMT
Hi Harsh,

Thanks for your detailed response! Now, the efficiency of my Yarn cluster
improved a lot after increasing the reducer number(mapreduce.job.reduces)
in mapred-site.xml. But I still have some questions about the way of Yarn
to execute MRv1 job:

1.In Hadoop 1.x, a job will be executed by map task and reduce task
together, with a typical process(map > shuffle > reduce). In Yarn, as I
know, a MRv1 job will be executed only by ApplicationMaster.
- Yarn could run multiple kinds of jobs(MR, MPI, ...), but, MRv1 job has
special execution process(map > shuffle > reduce) in Hadoop 1.x, and how
Yarn execute a MRv1 job? still include some special MR steps in Hadoop 1.x,
like map, sort, merge, combine and shuffle?
- Do the MRv1 parameters still work for Yarn? Like
mapreduce.task.io.sort.mb and mapreduce.map.sort.spill.percent?
- What's the general process for ApplicationMaster of Yarn to execute a
job?

2. In Hadoop 1.x, we can set the map/reduce slots by setting
'mapred.tasktracker.map.tasks.maximum' and
'mapred.tasktracker.reduce.tasks.maximum'
- For Yarn, above tow parameter do not work any more, as yarn uses
container instead, right?
- For Yarn, we can set the whole physical mem for a NodeManager using '
yarn.nodemanager.resource.memory-mb'. But how to set the default size of
physical mem of a container?
- How to set the maximum size of physical mem of a container? By the
parameter of 'mapred.child.java.opts'?

Thanks as always!

2013/6/9 Harsh J <harsh@cloudera.com>

> Hi Sam,
>
> > - How to know the container number? Why you say it will be 22 containers
> due to a 22 GB memory?
>
> The MR2's default configuration requests 1 GB resource each for Map
> and Reduce containers. It requests 1.5 GB for the AM container that
> runs the job, additionally. This is tunable using the properties
> Sandy's mentioned in his post.
>
> > - My machine has 32 GB memory, how many memory is proper to be assigned
> to containers?
>
> This is a general question. You may use the same process you took to
> decide optimal number of slots in MR1 to decide this here. Every
> container is a new JVM, and you're limited by the CPUs you have there
> (if not the memory). Either increase memory requests from jobs, to
> lower # of concurrent containers at a given time (runtime change), or
> lower NM's published memory resources to control the same (config
> change).
>
> > - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn',
> will other parameters for mapred-site.xml still work in yarn framework?
> Like 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent'
>
> Yes, all of these properties will still work. Old properties specific
> to JobTracker or TaskTracker (usually found as a keyword in the config
> name) will not apply anymore.
>
> On Sun, Jun 9, 2013 at 2:21 PM, sam liu <samliuhadoop@gmail.com> wrote:
> > Hi Harsh,
> >
> > According to above suggestions, I removed the duplication of setting, and
> > reduce the value of 'yarn.nodemanager.resource.cpu-cores',
> > 'yarn.nodemanager.vcores-pcores-ratio' and
> > 'yarn.nodemanager.resource.memory-mb' to 16, 8 and 12000. Ant then, the
> > efficiency improved about 18%.  I have questions:
> >
> > - How to know the container number? Why you say it will be 22 containers
> due
> > to a 22 GB memory?
> > - My machine has 32 GB memory, how many memory is proper to be assigned
> to
> > containers?
> > - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn',
> will
> > other parameters for mapred-site.xml still work in yarn framework? Like
> > 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent'
> >
> > Thanks!
> >
> >
> >
> > 2013/6/8 Harsh J <harsh@cloudera.com>
> >>
> >> Hey Sam,
> >>
> >> Did you get a chance to retry with Sandy's suggestions? The config
> >> appears to be asking NMs to use roughly 22 total containers (as
> >> opposed to 12 total tasks in MR1 config) due to a 22 GB memory
> >> resource. This could impact much, given the CPU is still the same for
> >> both test runs.
> >>
> >> On Fri, Jun 7, 2013 at 12:23 PM, Sandy Ryza <sandy.ryza@cloudera.com>
> >> wrote:
> >> > Hey Sam,
> >> >
> >> > Thanks for sharing your results.  I'm definitely curious about what's
> >> > causing the difference.
> >> >
> >> > A couple observations:
> >> > It looks like you've got yarn.nodemanager.resource.memory-mb in there
> >> > twice
> >> > with two different values.
> >> >
> >> > Your max JVM memory of 1000 MB is (dangerously?) close to the default
> >> > mapreduce.map/reduce.memory.mb of 1024 MB. Are any of your tasks
> getting
> >> > killed for running over resource limits?
> >> >
> >> > -Sandy
> >> >
> >> >
> >> > On Thu, Jun 6, 2013 at 10:21 PM, sam liu <samliuhadoop@gmail.com>
> wrote:
> >> >>
> >> >> The terasort execution log shows that reduce spent about 5.5 mins
> from
> >> >> 33%
> >> >> to 35% as below.
> >> >> 13/06/10 08:02:22 INFO mapreduce.Job:  map 100% reduce 31%
> >> >> 13/06/10 08:02:25 INFO mapreduce.Job:  map 100% reduce 32%
> >> >> 13/06/10 08:02:46 INFO mapreduce.Job:  map 100% reduce 33%
> >> >> 13/06/10 08:08:16 INFO mapreduce.Job:  map 100% reduce 35%
> >> >> 13/06/10 08:08:19 INFO mapreduce.Job:  map 100% reduce 40%
> >> >> 13/06/10 08:08:22 INFO mapreduce.Job:  map 100% reduce 43%
> >> >>
> >> >> Any way, below are my configurations for your reference. Thanks!
> >> >> (A) core-site.xml
> >> >> only define 'fs.default.name' and 'hadoop.tmp.dir'
> >> >>
> >> >> (B) hdfs-site.xml
> >> >>   <property>
> >> >>     <name>dfs.replication</name>
> >> >>     <value>1</value>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>dfs.name.dir</name>
> >> >>     <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_name_dir</value>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>dfs.data.dir</name>
> >> >>     <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_data_dir</value>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>dfs.block.size</name>
> >> >>     <value>134217728</value><!-- 128MB -->
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>dfs.namenode.handler.count</name>
> >> >>     <value>64</value>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>dfs.datanode.handler.count</name>
> >> >>     <value>10</value>
> >> >>   </property>
> >> >>
> >> >> (C) mapred-site.xml
> >> >>   <property>
> >> >>     <name>mapreduce.cluster.temp.dir</name>
> >> >>     <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduce_temp</value>
> >> >>     <description>No description</description>
> >> >>     <final>true</final>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>mapreduce.cluster.local.dir</name>
> >> >>
> >> >>
> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduce_local_dir</value>
> >> >>     <description>No description</description>
> >> >>     <final>true</final>
> >> >>   </property>
> >> >>
> >> >> <property>
> >> >>   <name>mapreduce.child.java.opts</name>
> >> >>   <value>-Xmx1000m</value>
> >> >> </property>
> >> >>
> >> >> <property>
> >> >>     <name>mapreduce.framework.name</name>
> >> >>     <value>yarn</value>
> >> >>    </property>
> >> >>
> >> >>  <property>
> >> >>     <name>mapreduce.tasktracker.map.tasks.maximum</name>
> >> >>     <value>8</value>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
> >> >>     <value>4</value>
> >> >>   </property>
> >> >>
> >> >>
> >> >>   <property>
> >> >>     <name>mapreduce.tasktracker.outofband.heartbeat</name>
> >> >>     <value>true</value>
> >> >>   </property>
> >> >>
> >> >> (D) yarn-site.xml
> >> >>  <property>
> >> >>     <name>yarn.resourcemanager.resource-tracker.address</name>
> >> >>     <value>node1:18025</value>
> >> >>     <description>host is the hostname of the resource manager
and
> >> >>     port is the port on which the NodeManagers contact the Resource
> >> >> Manager.
> >> >>     </description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <description>The address of the RM web application.</description>
> >> >>     <name>yarn.resourcemanager.webapp.address</name>
> >> >>     <value>node1:18088</value>
> >> >>   </property>
> >> >>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.resourcemanager.scheduler.address</name>
> >> >>     <value>node1:18030</value>
> >> >>     <description>host is the hostname of the resourcemanager
and port
> >> >> is
> >> >> the port
> >> >>     on which the Applications in the cluster talk to the Resource
> >> >> Manager.
> >> >>     </description>
> >> >>   </property>
> >> >>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.resourcemanager.address</name>
> >> >>     <value>node1:18040</value>
> >> >>     <description>the host is the hostname of the ResourceManager
and
> >> >> the
> >> >> port is the port on
> >> >>     which the clients can talk to the Resource Manager.
> </description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.nodemanager.local-dirs</name>
> >> >>
> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_local_dir</value>
> >> >>     <description>the local directories used by the
> >> >> nodemanager</description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.nodemanager.address</name>
> >> >>     <value>0.0.0.0:18050</value>
> >> >>     <description>the nodemanagers bind to this port</description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.nodemanager.resource.memory-mb</name>
> >> >>     <value>10240</value>
> >> >>     <description>the amount of memory on the NodeManager in
> >> >> GB</description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.nodemanager.remote-app-log-dir</name>
> >> >>
> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_app-logs</value>
> >> >>     <description>directory on hdfs where the application logs
are
> moved
> >> >> to
> >> >> </description>
> >> >>   </property>
> >> >>
> >> >>    <property>
> >> >>     <name>yarn.nodemanager.log-dirs</name>
> >> >>     <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_log</value>
> >> >>     <description>the directories used by Nodemanagers as log
> >> >> directories</description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.nodemanager.aux-services</name>
> >> >>     <value>mapreduce.shuffle</value>
> >> >>     <description>shuffle service that needs to be set for Map
Reduce
> to
> >> >> run </description>
> >> >>   </property>
> >> >>
> >> >>   <property>
> >> >>     <name>yarn.resourcemanager.client.thread-count</name>
> >> >>     <value>64</value>
> >> >>   </property>
> >> >>
> >> >>  <property>
> >> >>     <name>yarn.nodemanager.resource.cpu-cores</name>
> >> >>     <value>24</value>
> >> >>   </property>
> >> >>
> >> >> <property>
> >> >>     <name>yarn.nodemanager.vcores-pcores-ratio</name>
> >> >>     <value>3</value>
> >> >>   </property>
> >> >>
> >> >>  <property>
> >> >>     <name>yarn.nodemanager.resource.memory-mb</name>
> >> >>     <value>22000</value>
> >> >>   </property>
> >> >>
> >> >>  <property>
> >> >>     <name>yarn.nodemanager.vmem-pmem-ratio</name>
> >> >>     <value>2.1</value>
> >> >>   </property>
> >> >>
> >> >>
> >> >>
> >> >> 2013/6/7 Harsh J <harsh@cloudera.com>
> >> >>>
> >> >>> Not tuning configurations at all is wrong. YARN uses memory resource
> >> >>> based scheduling and hence MR2 would be requesting 1 GB minimum
by
> >> >>> default, causing, on base configs, to max out at 8 (due to 8 GB
NM
> >> >>> memory resource config) total containers. Do share your configs
as
> at
> >> >>> this point none of us can tell what it is.
> >> >>>
> >> >>> Obviously, it isn't our goal to make MR2 slower for users and to
not
> >> >>> care about such things :)
> >> >>>
> >> >>> On Fri, Jun 7, 2013 at 8:45 AM, sam liu <samliuhadoop@gmail.com>
> >> >>> wrote:
> >> >>> > At the begining, I just want to do a fast comparision of MRv1
and
> >> >>> > Yarn.
> >> >>> > But
> >> >>> > they have many differences, and to be fair for comparison
I did
> not
> >> >>> > tune
> >> >>> > their configurations at all.  So I got above test results.
After
> >> >>> > analyzing
> >> >>> > the test result, no doubt, I will configure them and do comparison
> >> >>> > again.
> >> >>> >
> >> >>> > Do you have any idea on current test result? I think, to compare
> >> >>> > with
> >> >>> > MRv1,
> >> >>> > Yarn is better on Map phase(teragen test), but worse on Reduce
> >> >>> > phase(terasort test).
> >> >>> > And any detailed suggestions/comments/materials on Yarn
> performance
> >> >>> > tunning?
> >> >>> >
> >> >>> > Thanks!
> >> >>> >
> >> >>> >
> >> >>> > 2013/6/7 Marcos Luis Ortiz Valmaseda <marcosluis2186@gmail.com>
> >> >>> >>
> >> >>> >> Why not to tune the configurations?
> >> >>> >> Both frameworks have many areas to tune:
> >> >>> >> - Combiners, Shuffle optimization, Block size, etc
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> 2013/6/6 sam liu <samliuhadoop@gmail.com>
> >> >>> >>>
> >> >>> >>> Hi Experts,
> >> >>> >>>
> >> >>> >>> We are thinking about whether to use Yarn or not in
the near
> >> >>> >>> future,
> >> >>> >>> and
> >> >>> >>> I ran teragen/terasort on Yarn and MRv1 for comprison.
> >> >>> >>>
> >> >>> >>> My env is three nodes cluster, and each node has similar
> hardware:
> >> >>> >>> 2
> >> >>> >>> cpu(4 core), 32 mem. Both Yarn and MRv1 cluster are
set on the
> >> >>> >>> same
> >> >>> >>> env. To
> >> >>> >>> be fair, I did not make any performance tuning on
their
> >> >>> >>> configurations, but
> >> >>> >>> use the default configuration values.
> >> >>> >>>
> >> >>> >>> Before testing, I think Yarn will be much better than
MRv1, if
> >> >>> >>> they
> >> >>> >>> all
> >> >>> >>> use default configuration, because Yarn is a better
framework
> than
> >> >>> >>> MRv1.
> >> >>> >>> However, the test result shows some differences:
> >> >>> >>>
> >> >>> >>> MRv1: Hadoop-1.1.1
> >> >>> >>> Yarn: Hadoop-2.0.4
> >> >>> >>>
> >> >>> >>> (A) Teragen: generate 10 GB data:
> >> >>> >>> - MRv1: 193 sec
> >> >>> >>> - Yarn: 69 sec
> >> >>> >>> Yarn is 2.8 times better than MRv1
> >> >>> >>>
> >> >>> >>> (B) Terasort: sort 10 GB data:
> >> >>> >>> - MRv1: 451 sec
> >> >>> >>> - Yarn: 1136 sec
> >> >>> >>> Yarn is 2.5 times worse than MRv1
> >> >>> >>>
> >> >>> >>> After a fast analysis, I think the direct cause might
be that
> Yarn
> >> >>> >>> is
> >> >>> >>> much faster than MRv1 on Map phase, but much worse
on Reduce
> >> >>> >>> phase.
> >> >>> >>>
> >> >>> >>> Here I have two questions:
> >> >>> >>> - Why my tests shows Yarn is worse than MRv1 for terasort?
> >> >>> >>> - What's the stratage for tuning Yarn performance?
Is any
> >> >>> >>> materials?
> >> >>> >>>
> >> >>> >>> Thanks!
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Marcos Ortiz Valmaseda
> >> >>> >> Product Manager at PDVSA
> >> >>> >> http://about.me/marcosortiz
> >> >>> >>
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Harsh J
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Mime
View raw message