hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar Vadali (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
Date Tue, 11 May 2010 22:51:41 GMT
Task Initialization should be delayed till when a job can be run
----------------------------------------------------------------

                 Key: MAPREDUCE-1783
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: contrib/fair-share
    Affects Versions: 0.20.1
            Reporter: Ramkumar Vadali


The FairScheduler task scheduler uses PoolManager to impose limits on the number of jobs that
can be running at a given time. However, jobs that are submitted are initiaiized immediately
by EagerTaskInitializationListener by calling JobInProgress.initTasks. This causes the job
split file to be read into memory. The split information is not needed until the number of
running jobs is less than the maximum specified. If the amount of split information is large,
this leads to unnecessary memory pressure on the Job Tracker.
To ease memory pressure, FairScheduler can use another implementation of JobInProgressListener
that is aware of PoolManager limits and can delay task initialization until the number of
running jobs is below the maximum.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message