hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-815) Investigate and fix the extremely large memory-footprint of JobTracker
Date Fri, 05 Jan 2007 07:03:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462418

Arun C Murthy commented on HADOOP-815:

> 1. The TaskInProgress.usableTaskIds should be removed and a new task id generated when
needed. That list has been bothering me for a while. *smile*
  Ok, coming up. :)

> 2. You replace a new style for loop with an old style for loop on TaskTracker.java line
597, which should stay a new style loop.
  Subtle change due to the fact that org.apache.hadoop.mapred.TaskTrackerStatus.getTaskReports
returns an 'Iterator'. Shud i fix the (public) api to return 'List<TaskStatus>' instead?

> 3. The trackerToMarkedTaskMap isn't synchronized by a lock, so has race conditions.
  The functions manipulating 'trackerToMarkedTaskMap' assume that the JobTracker itself is
locked on entry (much like existing 'removeTaskEntry'), would it help if I put in a comment
there? Or shud I explicitly mark the function as 'synchronized' ?

>On formatting, indentation is also four-spaces per level rather than the preferred two.

  Ok, I'll fix it. Spent time hitting the 'tab' key to ensure compliance with existing indentation
in some functions. *rueful smile*

  Related thought: should we open a new issue to track and ensure 'all' code is indented with
2 spaces instead of 4 (as in some places)? :)

> Investigate and fix the extremely large memory-footprint of JobTracker
> ----------------------------------------------------------------------
>                 Key: HADOOP-815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-815
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.1
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.0
>         Attachments: 150k_1199_774.nps, 75k_jobs.nps, HADOOP-815_20061220_1.patch, HADOOP-815_20061221_2.patch,
HADOOP-815_20061222_3.patch, HADOOP-815_20061230_4.patch, jt_memory_profiles.tgz
> The JobTracker's memory footprint seems excessively large, especially when many jobs
are submitted.
> Here is the 'top' output of a JobTracker which has scheduled ~1k jobs thus far:
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                
> 31877 arunc     19   0 2362m 261m  13m S 14.0 12.9  24:48.08 java  
> Clearly VIRTual memory of 2364Mb v/s 261Mb of RESident memory is symptomatic of this

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message