accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Sizing walog area
Date Thu, 24 Oct 2013 17:12:51 GMT
On Thu, Oct 24, 2013 at 12:58 PM, Terry P. <texpilot@gmail.com> wrote:

> Hi Christopher,
> Just to ensure I'm looking at the correct property, by "in-memory map size
> per tserver" are you referring to the property *tserver.memory.maps.max*in
> *accumulo-site.xml*?  If that's the case, I'm using 1GB for that
> property.
>
> I am running loggers on each tserver. Is the default that Accumulo writes
> to at least two loggers?  I see config item *
> logger.recovery.file.replication* is set to 2 (by default); is that what
> controls this?
>
> Digging around, I also see that logger.archive.replication is set to 2,
> and logger.archive is false.  What does logger.archive and
> logger.archive.replication do?  I find no mention of "archive" in the User
> Manual.
>

The archive option is something we use when testing.  We have a test suite
that continually writes linked list to Accumulo and a map reduce job that
can look for holes in the linked lists.  A hole indicates that Accumulo
lost data.   Having an archive off all of the walogs around is extremely
useful when trying to diagnose the problem.


>
> Many thanks Christopher, your help is always appreciated.
>
>
> On Wed, Oct 23, 2013 at 2:10 PM, Christopher <ctubbsii@apache.org> wrote:
>
>> The data in the write-ahead logs is needed until the tserver flushes
>> the in memory maps to disk. Assuming you have a logger running on
>> every tserver, and tservers write to at least two loggers, you should
>> ensure that the size of the disk area is *at least* two times as big
>> as your in-memory map size per tserver. I'd say 5x-10x the in-memory
>> map size is probably safe. So, if your tservers are running with 2GB
>> of memory, then a 10-20GB area is probably more than sufficient.
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Oct 23, 2013 at 1:02 PM, Terry P. <texpilot@gmail.com> wrote:
>> > Greetings all,
>> > For Accumulo 1.4 where write ahead logs are not yet stored in HDFS, does
>> > anyone have guidancewith respect to sizing the walog area?  What exactly
>> > triggers when write ahead logs get removed?  What might cause them to
>> hang
>> > around for an extended period of time (as in under abnormal
>> circumstances)?
>> >
>> > The system this applies to will see an ingest rate of approximately 2000
>> > docs per second averaging 1-2K each (broken out into 12 columns each, so
>> > 24,000 entries per second) across 6 tabletserver nodes.
>> >
>> > Thanks in advance,
>> > Terry
>>
>
>

Mime
View raw message