accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Out of memory when putting many rows in an Acc table
Date Wed, 01 Oct 2014 18:56:56 GMT
Or nofile (too). ulimit is your friend :)

Eric Newton wrote:
>     I realized this could be due to an inability by the JVM to create
>     additional native threads
>
>
> You may need to increase the nproc limit on your systems.
>
> -Eric
>
>
> On Wed, Oct 1, 2014 at 11:12 AM, Geoffry Roberts <threadedblue@gmail.com
> <mailto:threadedblue@gmail.com>> wrote:
>
>     Thanks for the response.
>
>     The only reason I was creating q new BatchWriter periodically was to
>     determine if BatchWriter was holding on to memory even after a
>     flush.   I had memory on my BatchWriterConfig set to 1M already.  I
>     am reading my RDB tables in pages of 10K rows.
>
>     Bumping up the JVM size didn't help,
>
>     I tried setting -XX:+HeapDumpOnOutOfMemoryError and when it did not
>     produce any output (hprof file). I realized this could be due to an
>     inability by the JVM to create additional native threads.
>
>     What I now think is the problem is not with Acc directly but hiding
>     out on the JDBC side.
>
>     Perhaps this is not an Acc issue at all but merely masquerading as
>     one.  We'll see.
>
>
>     On Tue, Sep 30, 2014 at 12:17 PM, Josh Elser <josh.elser@gmail.com
>     <mailto:josh.elser@gmail.com>> wrote:
>
>         You shouldn't have to create a new BatchWriter -- have you tried
>         reducing the amount of memory the BatchWriter will use? It keeps
>         a cache internally to try to do an amortization of Mutations to
>         send to a given tabletserver.
>
>         To limit this memory, use the
>         BatchWriterConfig#__setMaxMemory(long) method. By default, the
>         maxMemory value is set to 50MB. Reducing this may be enough to
>         hold less data in your client and give you some more head room.
>
>         Alternatively, you could give your client JVM some more heap :)
>
>
>         Geoffry Roberts wrote:
>
>             I am try to pump some data into Accumulo but I keep encountering
>
>             Exception in thread "Thrift Connection Pool Checker"
>             java.lang.OutOfMemoryError: Java heap space
>
>             at java.util.HashMap.__newValueIterator(HashMap.java:__971)
>
>             at java.util.HashMap$Values.__iterator(HashMap.java:1038)
>
>             at
>             org.apache.accumulo.core.__client.impl.__ThriftTransportPool$Closer.__closeConnections(__ThriftTransportPool.java:103)
>
>             at
>             org.apache.accumulo.core.__client.impl.__ThriftTransportPool$Closer.__run(ThriftTransportPool.java:__147)
>
>             at java.lang.Thread.run(Thread.__java:745)
>
>
>             I tried, as a work around, creating a new BatchWriter and
>             closing the
>             old one every ten thousand rows, but to no avail.  Data gets
>             written up
>             to the 200kth row, then the error.
>
>             I have a table of 8M rows in a RDB that I am pumping into
>             Acc via a
>             groovy script.  The rows are narrow, a short text field and
>             four floats.
>
>             I googled of course but nothing was helpful.  What can be done?
>
>             Thanks so much.
>
>             --
>             There are ways and there are ways,
>
>             Geoffry Roberts
>
>
>
>
>     --
>     There are ways and there are ways,
>
>     Geoffry Roberts
>
>

Mime
View raw message