hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jie Li <ji...@cs.duke.edu>
Subject Re: Hadoop 1.0.4 Performance Problem
Date Tue, 27 Nov 2012 15:08:56 GMT
Jon:

This is interesting and helpful! How did you figure out the cause? And how
much time did you spend? Could you share some experience of performance
diagnosis?

Jie

On Tuesday, November 27, 2012, Harsh J wrote:

> Hi Amit,
>
> The default scheduler is FIFO, and may not work for all forms of
> workloads. Read the multiple schedulers available to see if they have
> features that may benefit your workload:
>
> Capacity Scheduler:
> http://hadoop.apache.org/docs/stable/capacity_scheduler.html
> FairScheduler:
> http://hadoop.apache.org/docs/stable/fair_scheduler.html
>
> While there's a good overlap of features between them, there are a few
> differences that set them apart and make them each useful for
> different use-cases. If I had to summarize on some such differences,
> FairScheduler is better suited to SLA form of job execution situations
> due to its preemptive features (which make it useful in user and
> service mix scenarios), while CapacityScheduler provides
> manual-resource-request oriented scheduling for odd jobs with high
> memory workloads, etc. (which make it useful for running certain
> specific kind of jobs along side the regular ones).
>
> On Tue, Nov 27, 2012 at 3:51 PM, Amit Sela <amits@infolinks.com> wrote:
> > So this is a FairScheduler problem ?
> > We are using the default Hadoop scheduler. Is there a reason to use the
> Fair
> > Scheduler if most of the time we don't have more than 4 jobs running
> > simultaneously ?
> >
> >
> > On Tue, Nov 27, 2012 at 12:00 PM, Harsh J <harsh@cloudera.com> wrote:
> >>
> >> Hi Amit,
> >>
> >> He means the mapred.fairscheduler.assignmultiple FairScheduler
> >> property. It is true by default, which works well for most workloads
> >> if not benchmark style workloads. I would not usually trust that as a
> >> base perf. measure of everything that comes out of an upgrade.
> >>
> >> The other JIRA, MAPREDUCE-4451, has been resolved for 1.2.0.
> >>
> >> On Tue, Nov 27, 2012 at 3:20 PM, Amit Sela <amits@infolinks.com> wrote:
> >> > Hi Jon,
> >> >
> >> > I recently upgraded our cluster from Hadoop 0.20.3-append to Hadoop
> >> > 1.0.4
> >> > and I haven't noticed any performance issues. By  "multiple assignment
> >> > feature" do you mean speculative execution
> >> > (mapred.map.tasks.speculative.execution and
> >> > mapred.reduce.tasks.speculative.execution) ?
> >> >
> >> >
> >> > On Mon, Nov 26, 2012 at 11:49 PM, Jon Allen <jayayedev@gmail.com>
> wrote:
> >> >>
> >> >> Problem solved, but worth warning others about.
> >> >>
> >> >> Before the upgrade the reducers for the terasort process had been
> >> >> evenly
> >> >> distributed around the cluster - one per task tracker in turn,
> looping
> >> >> around the cluster until all tasks were allocated.  After the upgrade
> >> >> all
> >> >> reduce task had been submitted to small number of task trackers -
> >> >> submit
> >> >> tasks until the task tracker slots were full and then move onto the
> >> >> next
> >> >> task tracker.  Skewing the reducers like this quite clearly hit the
> >> >> benchmark performance.
> >> >>
> >> >> The reason for this turns out to be the fair scheduler rewrite
> >> >> (MAPREDUCE-2981) that appears to have subtly modified the behaviour
> of
> >> >> the
> >> >> assign multiple property. Previously this property caused a single
> map
> >> >> and a
> >> >> single reduce task to be allocated in a task tracker heartbeat
> (rather
> >> >> than
> >> >> the default of a map or a reduce).  After the upgrade it allocates
as
> >> >> many
> >> >> tasks as there are available task slots.  Turning off the multiple
> >> >> assignment feature returned the terasort to its pre-upgrade
> >> >> performance.
> >> >>
> >> >> I can see potential benefits to this change and need to think through
> >> >> the
> >> >> consequences to real world applications (though in practice we're
> >> >> likely to
> >> >> move away from fair scheduler due to MAPREDUCE-4451).  Investigating
> >> >> this
> >> >> has been a pain so to warn other user is there anywhere central that
> >> >> can be
> >> >> used to record upgrade gotchas like this?
> >> >>
> >> >>
> >> >> On Fri, Nov 23, 2012 at 12:02 PM, Jon Allen <jayayedev@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> We've just upgraded our cluster from Hadoop 0.20.203 to 1.0.4 and
> have
> >> >>> hit performance problems.  Before the upgrade a 15TB terasort took
> >> >>> about 45
> >> >>> minutes, afterwards it takes just over an hour.  Looking in more
> >> >>> detail it
> >> >>> appears the shuffle phase has increased from 20 minutes to 40
> minutes.
> >> >>> Does
> >> >>> anyone have any thoughts about what'--
> Harsh J
>
>

Mime
View raw message