hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop 1.0.4 Performance Problem
Date Tue, 27 Nov 2012 10:47:22 GMT
Hi Amit,

The default scheduler is FIFO, and may not work for all forms of
workloads. Read the multiple schedulers available to see if they have
features that may benefit your workload:

Capacity Scheduler: http://hadoop.apache.org/docs/stable/capacity_scheduler.html
FairScheduler:         http://hadoop.apache.org/docs/stable/fair_scheduler.html

While there's a good overlap of features between them, there are a few
differences that set them apart and make them each useful for
different use-cases. If I had to summarize on some such differences,
FairScheduler is better suited to SLA form of job execution situations
due to its preemptive features (which make it useful in user and
service mix scenarios), while CapacityScheduler provides
manual-resource-request oriented scheduling for odd jobs with high
memory workloads, etc. (which make it useful for running certain
specific kind of jobs along side the regular ones).

On Tue, Nov 27, 2012 at 3:51 PM, Amit Sela <amits@infolinks.com> wrote:
> So this is a FairScheduler problem ?
> We are using the default Hadoop scheduler. Is there a reason to use the Fair
> Scheduler if most of the time we don't have more than 4 jobs running
> simultaneously ?
>
>
> On Tue, Nov 27, 2012 at 12:00 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hi Amit,
>>
>> He means the mapred.fairscheduler.assignmultiple FairScheduler
>> property. It is true by default, which works well for most workloads
>> if not benchmark style workloads. I would not usually trust that as a
>> base perf. measure of everything that comes out of an upgrade.
>>
>> The other JIRA, MAPREDUCE-4451, has been resolved for 1.2.0.
>>
>> On Tue, Nov 27, 2012 at 3:20 PM, Amit Sela <amits@infolinks.com> wrote:
>> > Hi Jon,
>> >
>> > I recently upgraded our cluster from Hadoop 0.20.3-append to Hadoop
>> > 1.0.4
>> > and I haven't noticed any performance issues. By  "multiple assignment
>> > feature" do you mean speculative execution
>> > (mapred.map.tasks.speculative.execution and
>> > mapred.reduce.tasks.speculative.execution) ?
>> >
>> >
>> > On Mon, Nov 26, 2012 at 11:49 PM, Jon Allen <jayayedev@gmail.com> wrote:
>> >>
>> >> Problem solved, but worth warning others about.
>> >>
>> >> Before the upgrade the reducers for the terasort process had been
>> >> evenly
>> >> distributed around the cluster - one per task tracker in turn, looping
>> >> around the cluster until all tasks were allocated.  After the upgrade
>> >> all
>> >> reduce task had been submitted to small number of task trackers -
>> >> submit
>> >> tasks until the task tracker slots were full and then move onto the
>> >> next
>> >> task tracker.  Skewing the reducers like this quite clearly hit the
>> >> benchmark performance.
>> >>
>> >> The reason for this turns out to be the fair scheduler rewrite
>> >> (MAPREDUCE-2981) that appears to have subtly modified the behaviour of
>> >> the
>> >> assign multiple property. Previously this property caused a single map
>> >> and a
>> >> single reduce task to be allocated in a task tracker heartbeat (rather
>> >> than
>> >> the default of a map or a reduce).  After the upgrade it allocates as
>> >> many
>> >> tasks as there are available task slots.  Turning off the multiple
>> >> assignment feature returned the terasort to its pre-upgrade
>> >> performance.
>> >>
>> >> I can see potential benefits to this change and need to think through
>> >> the
>> >> consequences to real world applications (though in practice we're
>> >> likely to
>> >> move away from fair scheduler due to MAPREDUCE-4451).  Investigating
>> >> this
>> >> has been a pain so to warn other user is there anywhere central that
>> >> can be
>> >> used to record upgrade gotchas like this?
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:02 PM, Jon Allen <jayayedev@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> We've just upgraded our cluster from Hadoop 0.20.203 to 1.0.4 and have
>> >>> hit performance problems.  Before the upgrade a 15TB terasort took
>> >>> about 45
>> >>> minutes, afterwards it takes just over an hour.  Looking in more
>> >>> detail it
>> >>> appears the shuffle phase has increased from 20 minutes to 40 minutes.
>> >>> Does
>> >>> anyone have any thoughts about what's changed between these releases
>> >>> that
>> >>> may have caused this?
>> >>>
>> >>> The only change to the system has been to Hadoop.  We moved from a
>> >>> tarball install of 0.20.203 with all processes running as hadoop to
an
>> >>> RPM
>> >>> deployment of 1.0.4 with processes running as hdfs and mapred.
>> >>> Nothing else
>> >>> has changed.
>> >>>
>> >>> As a related question, we're still running with a configuration that
>> >>> was
>> >>> tuned for version 0.20.1. Are there any recommendations for tuning
>> >>> properties that have been introduced in recent versions that are worth
>> >>> investigating?
>> >>>
>> >>> Thanks,
>> >>> Jon
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Mime
View raw message