accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dylan Hutchison <dhutc...@cs.washington.edu>
Subject Re: Running low on memory and Zookeeper Session expired / disconnected
Date Fri, 11 Nov 2016 10:28:42 GMT
Oh no, do keep using the native maps.  I meant to say that the memory and
Zookeeper problems during ingest can be a result of *not *using native
maps.

The Zookeeper expire results from the tservers having to spend so much time
garbage collecting that the tservers can't respond to the Zookeeper
heartbeat, so Zookeeper expires their session and the tservers subsequently
kill themselves.

If you are using native maps and having these problems, then something else
may be the source.

On Fri, Nov 11, 2016 at 2:24 AM, Mario Pastorelli <
mario.pastorelli@teralytics.ch> wrote:

> Yes we are as they are supposed to be faster. Good to know they could be
> the problem but what would be the solution, disabling them? Also, I guess
> there isn't an easy way to see that they are the problem, right?
>
> On Fri, Nov 11, 2016 at 11:21 AM, Dylan Hutchison <
> dhutchis@cs.washington.edu> wrote:
>
>> Mario, are you using native in-memory maps?  I've seen these problems run
>> rampant under Java memory maps.
>>
>> On Fri, Nov 11, 2016 at 2:11 AM, Mario Pastorelli <
>> mario.pastorelli@teralytics.ch> wrote:
>>
>>> Hi all,
>>>
>>> I have two recurring errors with Accumulo in my cluster and I would like
>>> to know more about them. The first, usually happening at ingestion time
>>> when I write with the batch writers many records, is the "Running low on
>>> memory". We keep adding memory to Accumulo but this is a blind guess and I
>>> was wondering if there is a way to understand how much memory Accumulo
>>> would need considering the amount of data that will be written. Should we
>>> write slowly to Accumulo to avoid this? What is filling all the memory at
>>> ingestion time?
>>> Secondly, we have these zookeeper session expired and other zookeeper
>>> timeouts. Zookeeper on our cluster works quite well, we have many systems
>>> using it. How can I debug a "zookeeper session expired" in Accumulo?
>>>
>>> Thanks,
>>> Mario
>>>
>>> --
>>> Mario Pastorelli | TERALYTICS
>>>
>>> *software engineer*
>>>
>>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>>> phone: +41794381682
>>> email: mario.pastorelli@teralytics.ch
>>> www.teralytics.net
>>>
>>> Company registration number: CH-020.3.037.709-7 | Trade register Canton
>>> Zurich
>>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
>>> Yann de Vries
>>>
>>> This e-mail message contains confidential information which is for the
>>> sole attention and use of the intended recipient. Please notify us at once
>>> if you think that it may not be intended for you and delete it immediately.
>>>
>>
>>
>
>
> --
> Mario Pastorelli | TERALYTICS
>
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastorelli@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the
> sole attention and use of the intended recipient. Please notify us at once
> if you think that it may not be intended for you and delete it immediately.
>

Mime
View raw message