Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of eric.newton@gmail.com
 designates 209.85.216.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPnhrdts3KZo2LyZDAzhP+f4yhyZdUOSYy3oLFPQknn3YEhi+A@mail.gmail.com>
References: 
 <CAPnhrdu1ysX33eTgnna5-JUfAhWU4KJe70ZfzZ38y_KufHgEDg@mail.gmail.com>
	<CAL5zq9Zapp59d6g5KkF4+1HL+Y6EzH11T7MkaDjFND5tfnShFQ@mail.gmail.com>
	<CAPnhrdts3KZo2LyZDAzhP+f4yhyZdUOSYy3oLFPQknn3YEhi+A@mail.gmail.com>
Date: Thu, 24 Oct 2013 13:02:55 -0400
Message-ID: 
 <CADxc9BmHd8TO9ki2x5hChe__dDF-qJJJxfWfCRyWU1owVox0cg@mail.gmail.com>
Subject: Re: Sizing walog area
From: Eric Newton <eric.newton@gmail.com>
To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

logger.archive flag, when set,  copies the WALog files into HDFS
before they are removed from the local file system.  This is primarily
used for debugging when testing.  logger.recovery.file.replication is
a way to specify the replication of the archived WALog files in HDFS.

-Eric


On Thu, Oct 24, 2013 at 12:58 PM, Terry P. <texpilot@gmail.com> wrote:
> Hi Christopher,
> Just to ensure I'm looking at the correct property, by "in-memory map size
> per tserver" are you referring to the property tserver.memory.maps.max in
> accumulo-site.xml?  If that's the case, I'm using 1GB for that property.
>
> I am running loggers on each tserver. Is the default that Accumulo writes to
> at least two loggers?  I see config item logger.recovery.file.replication is
> set to 2 (by default); is that what controls this?
>
> Digging around, I also see that logger.archive.replication is set to 2, and
> logger.archive is false.  What does logger.archive and
> logger.archive.replication do?  I find no mention of "archive" in the User
> Manual.
>
> Many thanks Christopher, your help is always appreciated.
>
>
> On Wed, Oct 23, 2013 at 2:10 PM, Christopher <ctubbsii@apache.org> wrote:
>>
>> The data in the write-ahead logs is needed until the tserver flushes
>> the in memory maps to disk. Assuming you have a logger running on
>> every tserver, and tservers write to at least two loggers, you should
>> ensure that the size of the disk area is *at least* two times as big
>> as your in-memory map size per tserver. I'd say 5x-10x the in-memory
>> map size is probably safe. So, if your tservers are running with 2GB
>> of memory, then a 10-20GB area is probably more than sufficient.
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Oct 23, 2013 at 1:02 PM, Terry P. <texpilot@gmail.com> wrote:
>> > Greetings all,
>> > For Accumulo 1.4 where write ahead logs are not yet stored in HDFS, does
>> > anyone have guidancewith respect to sizing the walog area?  What exactly
>> > triggers when write ahead logs get removed?  What might cause them to
>> > hang
>> > around for an extended period of time (as in under abnormal
>> > circumstances)?
>> >
>> > The system this applies to will see an ingest rate of approximately 2000
>> > docs per second averaging 1-2K each (broken out into 12 columns each, so
>> > 24,000 entries per second) across 6 tabletserver nodes.
>> >
>> > Thanks in advance,
>> > Terry
>
>