hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4623) Running tasks are not maintained by JobInProgress if speculation is off
Date Fri, 21 Nov 2008 06:14:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649610#action_12649610
] 

Amar Kamat commented on HADOOP-4623:
------------------------------------

bq. That means each operation involves link manipulation and objection creation.
There is no object creation involved. Node is passed by the jobtracker which it maintains
per tracker/node. TIP is already created by the job as part of job-initialization. Only data
structure updates will happen using object references.

bq. Use a HashMap mapping nodes to fix sized arrays of tips. The fix size should be the number
of slots per node. 
Wouldnt this result into memory wastage as there might be many other jobs running in parallel
and the whole(expanded) _running-cache_ might not be used. How will you take care of holes
in the array? 

----
I think I should benchmark the patch again and see if the results are consistent. If yes then
we can go ahead with the patch and file an optimization jira for running-cache. Thoughts?

> Running tasks are not maintained by JobInProgress if speculation is off
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-4623
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4623
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>         Attachments: HADOOP-4623-v1.1.patch, HADOOP-4623-v1.2.patch
>
>
> {{JobInProgress}} doesnt maintain any structure for running tasks if speculation is turned
_off_.  {{getRunningMapCache()}} in {{JobInProgress}} exposes the running map cache. This
api returns an empty {{Map}} if speculation turned off. 
> _Usage_ :
> {{CapicityScheduler}} requires a list of running tasks for both speculated and non-speculated
jobs. See HADOOP-4558 to see how this issue affects {{CapacityScheduler}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message