Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of josh.elser@gmail.com
 designates 209.85.216.178 as permitted sender)
Message-ID: <53A9B455.8060200@gmail.com>
Date: Tue, 24 Jun 2014 13:24:37 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: Out Of Memory Error - Accumulo & Storm
References: 
 <CAAraj=Wo6bVkmqd9G0_68n3q3smnm+Ydg25As_+03oO77hywWA@mail.gmail.com>
In-Reply-To: 
 <CAAraj=Wo6bVkmqd9G0_68n3q3smnm+Ydg25As_+03oO77hywWA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Sivan,

This one is pretty straightforward when you decode it: you ran out of 
memory on your host.

Java threads will map to threads on the operating system and each of 
those OS thread requires memory. When the JVM tries to start a new 
thread, but can't because the OS can't allocate any more memory to make 
that new OS thread, you'll get this error.

Try freeing up some memory on your host (reduce max JVM heap sizes, turn 
down caches on BatchWriter, grow the memory if your host is a virtual 
machine, etc). It's a generally a good idea to leave a few hundred MB of 
memory available on your host for the OS to use.

Hope that helps.

On 6/24/14, 1:18 PM, Sivan Sasidharan wrote:
> Accumulo fails with the below error when running in the storm cluster.
> Please advice.
>
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:652)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:740)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:702)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:232)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:320)
> at
> org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:57)
> at
> com.ey.analytics.aap.realtime.core.bolts.adapters.RealTimeAccumuloTweetBolt.dataManipulate(RealTimeAccumuloTweetBolt.java:105)
> at
> com.ey.analytics.aap.realtime.core.bolts.RealTimeBolt.execute(RealTimeBolt.java:59)
> at
> backtype.storm.daemon.executor$fn__4050$tuple_action_fn__4052.invoke(executor.clj:566)
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__3976.invoke(executor.clj:345)
> at
> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
> at
> backtype.storm.daemon.executor$fn__4050$fn__4059$fn__4106.invoke(executor.clj:658)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Thread.java:662)
>
>
> Thanks
> Sivan Sasidaran
>