hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sela <am...@infolinks.com>
Subject Re: Hadoop 1.0.4 Performance Problem
Date Tue, 27 Nov 2012 10:21:31 GMT
So this is a FairScheduler problem ?
We are using the default Hadoop scheduler. Is there a reason to use the
Fair Scheduler if most of the time we don't have more than 4 jobs
running simultaneously ?

On Tue, Nov 27, 2012 at 12:00 PM, Harsh J <harsh@cloudera.com> wrote:

> Hi Amit,
>
> He means the mapred.fairscheduler.assignmultiple FairScheduler
> property. It is true by default, which works well for most workloads
> if not benchmark style workloads. I would not usually trust that as a
> base perf. measure of everything that comes out of an upgrade.
>
> The other JIRA, MAPREDUCE-4451, has been resolved for 1.2.0.
>
> On Tue, Nov 27, 2012 at 3:20 PM, Amit Sela <amits@infolinks.com> wrote:
> > Hi Jon,
> >
> > I recently upgraded our cluster from Hadoop 0.20.3-append to Hadoop 1.0.4
> > and I haven't noticed any performance issues. By  "multiple assignment
> > feature" do you mean speculative execution
> > (mapred.map.tasks.speculative.execution and
> > mapred.reduce.tasks.speculative.execution) ?
> >
> >
> > On Mon, Nov 26, 2012 at 11:49 PM, Jon Allen <jayayedev@gmail.com> wrote:
> >>
> >> Problem solved, but worth warning others about.
> >>
> >> Before the upgrade the reducers for the terasort process had been evenly
> >> distributed around the cluster - one per task tracker in turn, looping
> >> around the cluster until all tasks were allocated.  After the upgrade
> all
> >> reduce task had been submitted to small number of task trackers - submit
> >> tasks until the task tracker slots were full and then move onto the next
> >> task tracker.  Skewing the reducers like this quite clearly hit the
> >> benchmark performance.
> >>
> >> The reason for this turns out to be the fair scheduler rewrite
> >> (MAPREDUCE-2981) that appears to have subtly modified the behaviour of
> the
> >> assign multiple property. Previously this property caused a single map
> and a
> >> single reduce task to be allocated in a task tracker heartbeat (rather
> than
> >> the default of a map or a reduce).  After the upgrade it allocates as
> many
> >> tasks as there are available task slots.  Turning off the multiple
> >> assignment feature returned the terasort to its pre-upgrade performance.
> >>
> >> I can see potential benefits to this change and need to think through
> the
> >> consequences to real world applications (though in practice we're
> likely to
> >> move away from fair scheduler due to MAPREDUCE-4451).  Investigating
> this
> >> has been a pain so to warn other user is there anywhere central that
> can be
> >> used to record upgrade gotchas like this?
> >>
> >>
> >> On Fri, Nov 23, 2012 at 12:02 PM, Jon Allen <jayayedev@gmail.com>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> We've just upgraded our cluster from Hadoop 0.20.203 to 1.0.4 and have
> >>> hit performance problems.  Before the upgrade a 15TB terasort took
> about 45
> >>> minutes, afterwards it takes just over an hour.  Looking in more
> detail it
> >>> appears the shuffle phase has increased from 20 minutes to 40 minutes.
>  Does
> >>> anyone have any thoughts about what's changed between these releases
> that
> >>> may have caused this?
> >>>
> >>> The only change to the system has been to Hadoop.  We moved from a
> >>> tarball install of 0.20.203 with all processes running as hadoop to an
> RPM
> >>> deployment of 1.0.4 with processes running as hdfs and mapred.
>  Nothing else
> >>> has changed.
> >>>
> >>> As a related question, we're still running with a configuration that
> was
> >>> tuned for version 0.20.1. Are there any recommendations for tuning
> >>> properties that have been introduced in recent versions that are worth
> >>> investigating?
> >>>
> >>> Thanks,
> >>> Jon
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Mime
View raw message