accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Newton <eric.new...@gmail.com>
Subject Re: Out of memory when putting many rows in an Acc table
Date Wed, 01 Oct 2014 18:08:02 GMT
>
> I realized this could be due to an inability by the JVM to create
> additional native threads


You may need to increase the nproc limit on your systems.

-Eric


On Wed, Oct 1, 2014 at 11:12 AM, Geoffry Roberts <threadedblue@gmail.com>
wrote:

> Thanks for the response.
>
> The only reason I was creating q new BatchWriter periodically was to
> determine if BatchWriter was holding on to memory even after a flush.   I
> had memory on my BatchWriterConfig set to 1M already.  I am reading my RDB
> tables in pages of 10K rows.
>
> Bumping up the JVM size didn't help,
>
> I tried setting -XX:+HeapDumpOnOutOfMemoryError and when it did not
> produce any output (hprof file). I realized this could be due to an
> inability by the JVM to create additional native threads.
>
> What I now think is the problem is not with Acc directly but hiding out on
> the JDBC side.
>
> Perhaps this is not an Acc issue at all but merely masquerading as one.
> We'll see.
>
>
> On Tue, Sep 30, 2014 at 12:17 PM, Josh Elser <josh.elser@gmail.com> wrote:
>
>> You shouldn't have to create a new BatchWriter -- have you tried reducing
>> the amount of memory the BatchWriter will use? It keeps a cache internally
>> to try to do an amortization of Mutations to send to a given tabletserver.
>>
>> To limit this memory, use the BatchWriterConfig#setMaxMemory(long)
>> method. By default, the maxMemory value is set to 50MB. Reducing this may
>> be enough to hold less data in your client and give you some more head room.
>>
>> Alternatively, you could give your client JVM some more heap :)
>>
>>
>> Geoffry Roberts wrote:
>>
>>> I am try to pump some data into Accumulo but I keep encountering
>>>
>>> Exception in thread "Thrift Connection Pool Checker"
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at java.util.HashMap.newValueIterator(HashMap.java:971)
>>>
>>> at java.util.HashMap$Values.iterator(HashMap.java:1038)
>>>
>>> at
>>> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.
>>> closeConnections(ThriftTransportPool.java:103)
>>>
>>> at
>>> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.
>>> run(ThriftTransportPool.java:147)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> I tried, as a work around, creating a new BatchWriter and closing the
>>> old one every ten thousand rows, but to no avail.  Data gets written up
>>> to the 200kth row, then the error.
>>>
>>> I have a table of 8M rows in a RDB that I am pumping into Acc via a
>>> groovy script.  The rows are narrow, a short text field and four floats.
>>>
>>> I googled of course but nothing was helpful.  What can be done?
>>>
>>> Thanks so much.
>>>
>>> --
>>> There are ways and there are ways,
>>>
>>> Geoffry Roberts
>>>
>>
>
>
> --
> There are ways and there are ways,
>
> Geoffry Roberts
>

Mime
View raw message