cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Wee <peich...@gmail.com>
Subject Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread
Date Tue, 11 Nov 2014 05:15:00 GMT
Hi, thank you for response.

using 64bit and kernel 2.6.32-358.18.1.el6.x86_64.


# cat /proc/13405/limits
Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            10485760             unlimited            bytes

Max core file size        0                    unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             260000               260000
processes
Max open files            65536                65536                files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       255762               255762               signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us




Is the stack size and / or Max open files is sufficient? The rest fd pretty
much infinite.

Jason

On Tue, Nov 11, 2014 at 4:09 AM, Chris Lohfink <clohfink85@gmail.com> wrote:

> if your using 64 bit, check output of:
>
> cat /proc/{cassandra pid}/limits
>
> some older linux kernels wont work with above so if it doesnt exist check
> the ulimit -a output for the cassandra user. max processes per user may
> be your issue as well.
>
> ---
> Chris Lohfink
>
>
> On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <graham@vast.com>
> wrote:
>
>> First question are you running 32bit or 64bit… on 32bit you can easily
>> run out of virtual address space for thread stacks.
>>
>> On Nov 10, 2014, at 8:25 AM, Jason Wee <peichieh@gmail.com> wrote:
>>
>> Hello people, below is an extraction from cassandra system log.
>>
>> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
>> (line 139) Fatal exception in thread Thread[Thread-273,5,main]
>> java.lang.OutOfMemoryError: unable to create new native thread
>>         at java.lang.Thread.start0(Native Method)
>>         at java.lang.Thread.start(Thread.java:640)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
>>         at
>> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
>>         at
>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>>
>> Investigated into the call until the java native call,
>> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>>
>>   if (native_thread->osthread() == NULL) {
>>     // No one should hold a reference to the 'native_thread'.
>>     delete native_thread;
>>     if (JvmtiExport::should_post_resource_exhausted()) {
>>       JvmtiExport::post_resource_exhausted(
>>         JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
>> JVMTI_RESOURCE_EXHAUSTED_THREADS,
>>         "unable to create new native thread");
>>     }
>>     THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
>>               "unable to create new native thread");
>>   }
>>
>> Question. Is that out of memory error due to native os memory or java
>> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
>> max user processes 260000. open files capped at 65536
>>
>> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
>> and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>>
>> Thank you.
>>
>> Jason
>>
>>
>>
>

Mime
View raw message