hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sri Ramadasu (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-1900) the heartbeat and task event queries interval should be set dynamically by the JobTracker
Date Thu, 18 Oct 2007 10:07:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535840
] 

amareshwari edited comment on HADOOP-1900 at 10/18/07 3:07 AM:
----------------------------------------------------------------------------

bq. I wonder if instead we should just make it clusterSize/50+1? That way, small clusters
will get a heartbeat of just one second, which should make them more responsive.

Agreed.

bq. Now, MapEventsFetcherThread polls jobtracker for completed map tasks for every 5 secs
(MIN_POLL_INTERVAL). Shall we change polling interval also in the similar fashion as heartbeat
interval? But, here some reduce tasks could be idle for longer time.

We can have MapEventsFetcherThread polling jobtracker for completed map tasks  in the similar
fashion as heartbeat interval with MIN_POLL_INTERVAL=5secs and MAX_POLL_INTERVAL=30secs. 
And Whenever tasktracker finds there are no mapevents and reduce task is waiting for the map
events, it will wakeup the thread to fetch map events from job tracker.

      was (Author: amareshwari):
    bq. I wonder if instead we should just make it clusterSize/50+1? That way, small clusters
will get a heartbeat of just one second, which should make them more responsive.

Agreed.

bq. Now, MapEventsFetcherThread polls jobtracker for completed map tasks for every 5 secs
(MIN_POLL_INTERVAL). Shall we change polling interval also in the similar fashion as heartbeat
interval? But, here some reduce tasks could be idle for longer time.

We can have MapEventsFetcherThread polling jobtracker for completed map tasks  in the similar
fashion as heartbeat interval with MIN_POLL_INTERVAL=5secs and MIN_POLL_INTERVAL_MAX=30secs.
 And Whenever tasktracker finds there are no mapevents and reduce task is waiting for the
map events, it will wakeup the thread to fetch map events from job tracker.
  
> the heartbeat and task event queries interval should be set dynamically by the JobTracker
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1900
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>
> The JobTracker should scale the intervals that the TaskTrackers use to contact it dynamically,
based on how the busy it is and the size of the cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message