accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Newton <eric.new...@gmail.com>
Subject Re: Write ahead log growth and reduction
Date Wed, 11 Apr 2012 15:12:26 GMT
Logs are only put into HDFS during a recovery.

Flush removes references to WALs, and the accumulo gc will ask the loggers
to delete them when there are no references to them.

-Eric

On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane <kkane.list@gmail.com>wrote:

>
>
> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner <keith@deenlo.com> wrote:
>
>> How big is the partition?  Are the same number of logger servers
>> running as tablet servers?
>>
>>
>> You can scan the metadata table to look for tablets that have alot of
>> write-ahead logs. I think the command below will show you how many
>> write-ahead logs each tablet has.  Look for any tablets that have too
>> many. I think it should sort the tablets with the most tablets to the
>> top, but not positive.
>>
>>   ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c
>> log' | cut -f 1 -d ' ' | uniq -c  | sort -r -n
>>
>> I think the following command will show you how many active log each
>> logger has.  This should be even.
>>
>>   ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c
>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 | sort | uniq -c
>>
>> You can use the "flush -p" command in the shell to force data in
>> memory to disk and stop referencing write-ahead logs.  Maybe execute
>> the commands above before and after flushing.
>>
>> Keith
>>
>>
>>
>
>
> Thanks for the replies.  I read about the flush command in the docs but
> didn't make a connection between "memory" to the write ahead logs.  Is
> that correct?  Flush writes write ahead log data to hdfs?
>
> Thanks!
>
> -Kris
>

Mime
View raw message