hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rahul k singh (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5214) ConcurrentModificationException in FairScheduler.getTotalSlots
Date Tue, 17 Feb 2009 11:40:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674179#action_12674179

rahul k singh commented on HADOOP-5214:

Had thought about following approaches in mind.

1. Generate the local copy of taskTrackers and go over them to find out total slots.
2. Synchronizing the access points of taskTrackers().values esp making sure iterators are
in synchronizing block.
3. Using ClusterStatus to fetch the required information , ClusterStatus already calculates
the MaxMapTasks and MaxReduceTasks at cluster level ( cluster's maximum capacity for running
map or reduce tasks).

took the 3rd approach.

changed the synchronzation of update() method to more granular level , this allows us to directly
get lock over JobTracker to fetch the clusterStatus, instead of taking JobTracker lock and
then scheduler lock.
Currently update() method is like below
 protected void update() {
    ClusterStatus clusterStatus = taskTrackerManager.getClusterStatus();

getTotalSlots now takes additional parameter clusterStatus.
PS : We have made getTotalSlots private , as we couldnt find any usage of this method outside
the class and reason why we would need to make this method public.

updateFairShare() method also takes clusterStatus as parameter.

PS : Matei can you look into the patch.

> ConcurrentModificationException in FairScheduler.getTotalSlots
> --------------------------------------------------------------
>                 Key: HADOOP-5214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5214
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/fair-share
>            Reporter: Vinod K V
>            Assignee: rahul k singh

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message