hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
Date Fri, 21 May 2010 20:14:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870141#action_12870141
] 

Hadoop QA commented on MAPREDUCE-1783:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12445178/submit-mapreduce-1783.patch
  against trunk revision 946955.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/console

This message is automatically generated.

> Task Initialization should be delayed till when a job can be run
> ----------------------------------------------------------------
>
>                 Key: MAPREDUCE-1783
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Ramkumar Vadali
>             Fix For: 0.22.0
>
>         Attachments: 0001-Pool-aware-job-initialization.patch, 0001-Pool-aware-job-initialization.patch.1,
submit-mapreduce-1783.patch
>
>
> The FairScheduler task scheduler uses PoolManager to impose limits on the number of jobs
that can be running at a given time. However, jobs that are submitted are initiaiized immediately
by EagerTaskInitializationListener by calling JobInProgress.initTasks. This causes the job
split file to be read into memory. The split information is not needed until the number of
running jobs is less than the maximum specified. If the amount of split information is large,
this leads to unnecessary memory pressure on the Job Tracker.
> To ease memory pressure, FairScheduler can use another implementation of JobInProgressListener
that is aware of PoolManager limits and can delay task initialization until the number of
running jobs is below the maximum.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message