hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (MAPREDUCE-44) Hang JobTracker, running out of memory
Date Mon, 27 Sep 2010 05:05:33 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-44?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amareshwari Sriramadasu resolved MAPREDUCE-44.
----------------------------------------------

      Assignee:     (was: Amareshwari Sriramadasu)
    Resolution: Not A Problem

Closing old ticket.

> Hang JobTracker, running out of memory
> --------------------------------------
>
>                 Key: MAPREDUCE-44
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-44
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>
> This may be expected.
> Hang JobTracker with 1G heapsize, top showed 99% cpu. 
> Ran about 80 jobs.  Each with 2500 mappers 200 reducers.  They finish quite fast.  3-4
mins avg per job.
> (200k tasks)
> How much memory does JobTracker use for 'completed'  (but not expired) jobs ?
> jmap -heap showed 
> {noformat} 
> ...
> PS Old Generation
>    capacity = 932118528 (888.9375MB)
>    used     = 932118528 (888.9375MB)
> ...
> {noformat} 
> jmap -histo showed 
> {noformat} 
> num   #instances    #bytes  class name
> --------------------------------------
>   1:   3974182   355869992  [C
>   2:   5216606   125198544  java.lang.String
>   3:   2238560   107450880  java.util.TreeMap
>   4:    463206   101673488  [B
>   5:   1979995    63359840  java.util.TreeMap$Entry
>   6:    248400    35769600  org.apache.hadoop.mapred.TaskInProgress
>   7:    308803    30898112  [Ljava.lang.Object;
>   8:    978240    23477760  org.apache.hadoop.mapred.Counters$CounterRec
>   9:    249876    19990080  org.apache.hadoop.mapred.TaskStatus
>  10:    248836    19906880  java.net.URI
>  11:    230337    16584264  org.apache.hadoop.mapred.MapTask
> ...
> {noformat} 
> Log showing many heartbeat discarded messages
> {noformat} 
> 2007-10-30 22:55:46,912 WARN org.apache.hadoop.ipc.Server: IPC Server handler 6 on 58567,
call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1afb9c9, false, true, 3942) from
99.99.99.99:9999 discarded for being too old (2578616)
> {noformat} 
> Is the solution either to increase the jobtracker heapsize or set shorter 'mapred.userlog.retain.hours'
 ?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message