accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Parker <parker20...@gmail.com>
Subject Re: Accumulo 1.4 Memory Issues
Date Sat, 04 Aug 2012 20:37:46 GMT
I did write a set of tests to randomly insert, delete, update (delete and
insert again) records, which had no issues into 100K operations and using
large, random column family keys.

I finally tracked down the error to a BatchScanner that wasn't being
closed, which consumed every ounce client memory when hundreds of these
were waiting around for something to do. So everything is well with the
world again. Thanks for all the suggestions.


On Thu, Aug 2, 2012 at 1:44 PM, John Vines <vines@apache.org> wrote:

> What dies, the tserver or the client?
>
> If it's the client, then you either need to decrease the size of your
> batchwriter when you create it or you need to up the heap space for your
> client code.
>
> If it's the tserver, make sure you A. picked a configuration setup who's
> footprint is large enough for your system + other things running, including
> zookeeper, hadoop, and your client code.
>
> John
>
>
> On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <parker20121@gmail.com> wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <marc@accumulo.net> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <parker20121@gmail.com>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>

Mime
View raw message