hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang
Date Thu, 01 Dec 2016 05:23:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710939#comment-15710939
] 

Brahma Reddy Battula commented on HDFS-11192:
---------------------------------------------

bq.Was it hitting its ulimit on # of threads/processes? 
Yes, Some other process creating lot of threads which leads to cross user limit.
bq.A potential improvement will be to terminate NN if this happens.

To call terminate from code, there was no exception handling done for this part of the code.

To terminate NN on OOM, java VM option {{-XX:OnOutOfMemoryError="kill -9 %p"}} was passed,
even that didnt take effect.


> OOM during Quota Initialization lead to Namenode hang
> -----------------------------------------------------
>
>                 Key: HDFS-11192
>                 URL: https://issues.apache.org/jira/browse/HDFS-11192
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Brahma Reddy Battula
>            Assignee: Brahma Reddy Battula
>         Attachments: namenodeThreadDump.out
>
>
> AFAIK ,In RecurisveTask Execution, When ForkjoinThreadpool's thread dies or not able
to create,it will not notify the parent.Parent still waiting for the notify call..that's not
timed waiting also.
>  *Trace from Namenode log* 
> {noformat}
> Exception in thread "ForkJoinPool-1-worker-2" Exception in thread "ForkJoinPool-1-worker-3"
java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:714)
>         at java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
>         at java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
>         at java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
>         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:714)
>         at java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
>         at java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
>         at java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
>         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message