accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Newton <eric.new...@gmail.com>
Subject Re: Write ahead log growth and reduction
Date Wed, 11 Apr 2012 17:00:24 GMT
Also you will need to restart.

What version of accumulo are you running?  This problem should have been
fixed with ACCUMULO-243.

-Eric

On Wed, Apr 11, 2012 at 12:28 PM, Eric Newton <eric.newton@gmail.com> wrote:

> in conf/accumulo-site.xml, add this:
>
>     <property>
>       <name>trace.password</name>
>       <value>your-root-password-here</value>
>     </property>
>
> -Eric
>
> On Wed, Apr 11, 2012 at 12:14 PM, Kristopher Kane <kkane.list@gmail.com>wrote:
>
>> I know I haven't answered some of the previous questions to me, but, the
>> GC talk brought me to the source of the problem.  our WAL GC job is not
>> successfully running.  The GC logs state a bad username/password, so, will
>> investigate that.
>>
>> Thanks for the help!
>>
>> -Kris
>>
>> On Wed, Apr 11, 2012 at 11:18 AM, Keith Turner <keith@deenlo.com> wrote:
>>
>>> When data is written to the accumulo its written to memory and the
>>> write ahead logs.  The data in memory is sorted and the data in the
>>> write ahead logs is written asis (unsorted).  When the data in memory
>>> is flushed to HDFS, the write ahead logs that also contain that data
>>> are no longer needed.
>>>
>>> On Wed, Apr 11, 2012 at 11:12 AM, Eric Newton <eric.newton@gmail.com>
>>> wrote:
>>> > Logs are only put into HDFS during a recovery.
>>> >
>>> > Flush removes references to WALs, and the accumulo gc will ask the
>>> loggers
>>> > to delete them when there are no references to them.
>>> >
>>> > -Eric
>>> >
>>> >
>>> > On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane <
>>> kkane.list@gmail.com>
>>> > wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner <keith@deenlo.com>
>>> wrote:
>>> >>>
>>> >>> How big is the partition?  Are the same number of logger servers
>>> >>> running as tablet servers?
>>> >>>
>>> >>>
>>> >>> You can scan the metadata table to look for tablets that have alot
of
>>> >>> write-ahead logs. I think the command below will show you how many
>>> >>> write-ahead logs each tablet has.  Look for any tablets that have
too
>>> >>> many. I think it should sort the tablets with the most tablets to
the
>>> >>> top, but not positive.
>>> >>>
>>> >>>   ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c
>>> >>> log' | cut -f 1 -d ' ' | uniq -c  | sort -r -n
>>> >>>
>>> >>> I think the following command will show you how many active log
each
>>> >>> logger has.  This should be even.
>>> >>>
>>> >>>   ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c
>>> >>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 | sort | uniq -c
>>> >>>
>>> >>> You can use the "flush -p" command in the shell to force data in
>>> >>> memory to disk and stop referencing write-ahead logs.  Maybe execute
>>> >>> the commands above before and after flushing.
>>> >>>
>>> >>> Keith
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> Thanks for the replies.  I read about the flush command in the docs
>>> but
>>> >> didn't make a connection between "memory" to the write ahead logs. 
Is
>>> >> that correct?  Flush writes write ahead log data to hdfs?
>>> >>
>>> >> Thanks!
>>> >>
>>> >> -Kris
>>> >
>>> >
>>>
>>
>>
>

Mime
View raw message