hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-815) Investigate and fix the extremely large memory-footprint of JobTracker
Date Fri, 05 Jan 2007 23:10:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462635
] 

Owen O'Malley commented on HADOOP-815:
--------------------------------------

Here are the paths that don't lock the JobTracker as far as i can see...

ExpireLaunchingTask.run -> 
  JobInProgress.failedTask (line 670) ->
  JobInProgress.updateTaskStatus ->
  JobInProgress.failedTask (line 592)
  JobTracker.markCompletedTaskAttempt

and:

ExpireTrackers.run ->
  JobTracker.lostTaskTracker ->
  JobInProgress.failedTask (line 670) -> from above

and a couple of similar variations. I think we are heading the wrong way with this, because
if it is this hard to verify that it is right, we will not be able to keep it from breaking
when we change the code later.

> Investigate and fix the extremely large memory-footprint of JobTracker
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-815
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.1
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.1
>
>         Attachments: 150k_1199_774.nps, 75k_jobs.nps, HADOOP-815_20061220_1.patch, HADOOP-815_20061221_2.patch,
HADOOP-815_20061222_3.patch, HADOOP-815_20061230_4.patch, HADOOP-815_20070105_5.patch, HADOOP-815_20070106_6.patch,
jt_memory_profiles.tgz
>
>
> The JobTracker's memory footprint seems excessively large, especially when many jobs
are submitted.
> Here is the 'top' output of a JobTracker which has scheduled ~1k jobs thus far:
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                
                                                                                    
> 31877 arunc     19   0 2362m 261m  13m S 14.0 12.9  24:48.08 java  
> Clearly VIRTual memory of 2364Mb v/s 261Mb of RESident memory is symptomatic of this
issue...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message