flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cochran, David" <david.coch...@bsee.gov>
Subject java.lang.OutOfMemoryError: unable to create new native thread
Date Tue, 26 Nov 2013 20:20:08 GMT
I've got a pretty good sized box collecting logs for a number of sources
(about a dozen or so).
Actually two instances were running on this box (one production and the
other a testing environment)

After adding a dozen more log files i'm running into this error (87 logs
for PROD and 70 or so for Testing)

Nov 26, 2013 10:25:27 AM
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
WARNING: Failed to accept a connection.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
 at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.start(AbstractNioWorker.java:160)
 at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.register(AbstractNioWorker.java:131)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.registerAcceptedChannel(NioServerSocketPipelineSink.java:269)
 at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:231)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I have been able to confirm that they are not bumping into the OS ulimit,
and increasing the sizes of the JVM does not make this go away
(JAVA_OPTS="-Xms1g -Xmx6g -Dcom.sun.management.jmxremot").  I ended up
having to shut down the testing environment instance so as not to lose any
more production logs.

The box is 16 CPU with 16G RAM using local disks, running RHEL5.x

Anytime process threads exceed ~930 (the results of  ps -eLF -u flume | wc
-l )the above error appears...if allowed to continue ultimately crashes the
box.

Any ideas on resolving this issue or have I run into a thread limit that
cannot be gotten around without adding another server?

Thanks,
Dave

Mime
View raw message