hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Simultaneous jobs and map/reduce sharing
Date Fri, 25 Apr 2008 00:10:23 GMT

I'm trying to run multiple jobs on the same cluster and get them to run simultaneously.  I
have them running simultaneously "somewhat", but have some questions (didn't find answers
in FAQ nor Wiki).

I start 2 jobs with a short (10 sec) pause between them. Job 1 quickly grabs all available
map tasks and "hogs" them.  Consequently, Job 2 has all its map tasks in pending mode until
Job 1 gets closer to the end and starts freeing up map tasks.

Cluster size = 4 nodes
Cluster Map Task Capacity = 16
Cluster Reduce Task Capacity = 8
mapred.tasktracker.map.tasks.maximum = 4
mapred.tasktracker.reduce.tasks.maximum = 2
mapred.map.tasks = 23
mapred.reduce.tasks = 11
mapred.speculative.execution = false

Job 1:
Map Total = 21
Reduce Total = 11

Job 2:
Map Total = 63
Reduce Total = 23

When Job 1 start, it quickly grabs all 16 map tasks (the Cluster Map Task Capacity) and only
several hours later, when it completes 6 of its 21 tasks (21-6=15, which is < 16), it starts
freeing up map slots for Job 2.  The same thing happens in the reduce phase.

What I'd like is to find a way to control how much each job gets and thus schedule them better.
 I believe I could change the number of "Map Total" for Job 1, so that it is < Cluster
Map Task Capacity, so that Job 2 can get at least one map slot right away, but then that Job
1 will take longer.  

If it matters, Job 1 and Job 2 are very different - Job 1 is network intensive (Nutch fetcher)
and Job 2 is CPU and disk IO intensive (Nutch generate job).  If I start them separately with
whole cluster dedicated to a single running job, then Job 1 finishes in about 10 hours, and
Job 2 finished is about 1.5 hours.  I was hoping to start the slow Job 1 and, while it's running,
maximize the use of the CPU by running and completing several Job 2 instances.

Are there other, better options?

Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

View raw message