hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-815) Investigate and fix the extremely large memory-footprint of JobTracker
Date Sun, 31 Dec 2006 07:19:22 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-815?page=all ]

Arun C Murthy updated HADOOP-815:
---------------------------------

    Attachment: HADOOP-815_20061230_4.patch
                jt_memory_profiles.tgz

Ok, here is a reasonably well-tested patch...

List of changes:
  a) Fixed HADOOP-740 i.e. ensure task entries are cleaned up on completion.
  b) Fixed HADOOP-787 i.e. ensure we keep only 100jobs per user; rest are available via jobhistroy
anyway.
  c) Fixed both JobTracker & TaskTracker to ensure lost status-updates/heartbeatResponses
due to lost rpcs are resent by both TaskTracker & JobTracker; and also that the JobTracker
can detect that duplicate 'TaskTrackerStatus' updates and ignore them, which otherwise are
fatal.
  d) Some miscellaneous fixes like using ArrayList instead of TreeSet and array for 'usableTaskIds'
in TaskInProgress.java
  
  Results:
  Currently after running smallJobsBenchmark with 750 jobs each with 300 maps & 2 reduces
(i.e. total of ~225,000 tasks) the memory footprint of the JobTracker is ~1.5Gb after 'RETIRE_JOB_INTERVAL'
(which I suspect also leads to degeneration of JT's performance as in HADOOP-843 since each
of the JT's datastructures are extremely bloated leading to sluggishness). With this patch
the memory-footprint is down to ~150MB after 'RETIRE_JOB_INTERVAL', yes, that's 150Mb! :)
(and seems to solve HADOOP-843 too).

  Appreciate any feedback...


> Investigate and fix the extremely large memory-footprint of JobTracker
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-815
>                 URL: http://issues.apache.org/jira/browse/HADOOP-815
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.1
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.0
>
>         Attachments: 150k_1199_774.nps, 75k_jobs.nps, HADOOP-815_20061220_1.patch, HADOOP-815_20061221_2.patch,
HADOOP-815_20061222_3.patch, HADOOP-815_20061230_4.patch, jt_memory_profiles.tgz
>
>
> The JobTracker's memory footprint seems excessively large, especially when many jobs
are submitted.
> Here is the 'top' output of a JobTracker which has scheduled ~1k jobs thus far:
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                
                                                                                    
> 31877 arunc     19   0 2362m 261m  13m S 14.0 12.9  24:48.08 java  
> Clearly VIRTual memory of 2364Mb v/s 261Mb of RESident memory is symptomatic of this
issue...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message