hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3116) Investigate whether the array in the JobInProgress that holds TIP references can be removed
Date Wed, 07 May 2008 06:21:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594790#action_12594790
] 

Amar Kamat commented on HADOOP-3116:
------------------------------------

One simple thing we can do is to use a {{MAP}} from _attempt-id_ to _TIP_ instead of an array.

> Investigate whether the array in the JobInProgress that holds TIP references can be removed
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3116
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3116
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Devaraj Das
>             Fix For: 0.18.0
>
>
> The array, in JobInProgress, that holds the references to TIPs is required now only to
serve the clients (web UI via JSPs and JobClient APIs, that traverse the array and create
the output). The array can now be removed since HADOOP-2119 introduces some datastructures
for running/non-running TIPs and we could probably add some more if required (e.g., for completed
TIPs). That way, we will gain when we have large jobs (many tasks) in the sense that we don't
have to go through the big array every time a client request is made. We could instead iterate
over the datastructures. Also, we should do these traversals without locking the JobTracker
to avoid cases where long traversal leads to loss of heartbeats, etc. Some staleness at the
client side (in favor of improving the JobTracker's performance) is probably okay.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message