hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4662) JobHistoryFilesManager thread pool never expands
Date Mon, 24 Sep 2012 17:08:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461918#comment-13461918

Kihwal Lee commented on MAPREDUCE-4662:

As you pointed out, the core thread timeout only takes effect when there is no work (i.e.
no job completion). I took the heap dump of JT to see the overhead of these extra threads.
In terms of memory, counting HashMapEntry, Thread, Worker and thread local objects that are
unique to a thread, it is well under 1KB per thread. I suspect stack and other supporting
system data structures take more memory. So, having them around all the time doesn't seem
to add much resource overhead.  Probably starting and stopping them frequently will create
more overhead. 
> JobHistoryFilesManager thread pool never expands
> ------------------------------------------------
>                 Key: MAPREDUCE-4662
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4662
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>    Affects Versions: 1.0.2
>            Reporter: Thomas Graves
>            Assignee: Kihwal Lee
>         Attachments: mapreduce-4662.branch-1.patch, mapreduce-4662.branch-1.patch
> The job history file manager creates a threadpool with core size 1 thread, max pool size
3.   It never goes beyond 1 thread though because its using a LinkedBlockingQueue which doesn't
have a max size. 
>     void start() {
>       executor = new ThreadPoolExecutor(1, 3, 1,
>           TimeUnit.HOURS, new LinkedBlockingQueue<Runnable>());
>     }
> According to the ThreadPoolExecutor java doc page it only increases the number of threads
when the queue is full. Since the queue we are using has no max size it never fills up and
we never get more then 1 thread. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message