accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristopher Kane <kkane.l...@gmail.com>
Subject Re: Write ahead log growth and reduction
Date Mon, 30 Apr 2012 03:49:37 GMT
I wanted to get back to the group with results.  We upgraded to 1.3.4 and
the WA GC worked after start up.  Unfortunately, I don't know if it was the
restart or the upgrade that mattered as the JIRA references the fix in
1.3.6.

Thanks for all of the help.

-Kris

On Thu, Apr 12, 2012 at 11:02 AM, Eric Newton <eric.newton@gmail.com> wrote:

> Running the tracer is optional.  Just make an empty tracers file in the
> conf directory and restart.
>
> -Eric
>
> On Thu, Apr 12, 2012 at 10:21 AM, Kristopher Kane <kkane.list@gmail.com>wrote:
>
>> Eric, thanks for the help.
>>
>> This is on 1.3.2, looks like we will be able to go to 1.3.4 very
>> shortly.  From the below JIRA, it looks like it is fixed in 1.3.6.  I will
>> send this up as I'm not an approving authority on installed/running
>> versions and will also recommend the trace password in the site.xml.
>>
>> Once we upgrade to the minor version, I will report back on whether the
>> WAL GC runs in that version.
>>
>> -Kris
>>
>> On Wed, Apr 11, 2012 at 1:00 PM, Eric Newton <eric.newton@gmail.com>wrote:
>>
>>> Also you will need to restart.
>>>
>>> What version of accumulo are you running?  This problem should have been
>>> fixed with ACCUMULO-243.
>>>
>>> -Eric
>>>
>>> On Wed, Apr 11, 2012 at 12:28 PM, Eric Newton <eric.newton@gmail.com>wrote:
>>>
>>>> in conf/accumulo-site.xml, add this:
>>>>
>>>>     <property>
>>>>       <name>trace.password</name>
>>>>       <value>your-root-password-here</value>
>>>>     </property>
>>>>
>>>> -Eric
>>>>
>>>> On Wed, Apr 11, 2012 at 12:14 PM, Kristopher Kane <kkane.list@gmail.com
>>>> > wrote:
>>>>
>>>>> I know I haven't answered some of the previous questions to me, but,
>>>>> the GC talk brought me to the source of the problem.  our WAL GC job
is not
>>>>> successfully running.  The GC logs state a bad username/password, so,
will
>>>>> investigate that.
>>>>>
>>>>> Thanks for the help!
>>>>>
>>>>> -Kris
>>>>>
>>>>> On Wed, Apr 11, 2012 at 11:18 AM, Keith Turner <keith@deenlo.com>wrote:
>>>>>
>>>>>> When data is written to the accumulo its written to memory and the
>>>>>> write ahead logs.  The data in memory is sorted and the data in the
>>>>>> write ahead logs is written asis (unsorted).  When the data in memory
>>>>>> is flushed to HDFS, the write ahead logs that also contain that data
>>>>>> are no longer needed.
>>>>>>
>>>>>> On Wed, Apr 11, 2012 at 11:12 AM, Eric Newton <eric.newton@gmail.com>
>>>>>> wrote:
>>>>>> > Logs are only put into HDFS during a recovery.
>>>>>> >
>>>>>> > Flush removes references to WALs, and the accumulo gc will ask
the
>>>>>> loggers
>>>>>> > to delete them when there are no references to them.
>>>>>> >
>>>>>> > -Eric
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane <
>>>>>> kkane.list@gmail.com>
>>>>>> > wrote:
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner <keith@deenlo.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> How big is the partition?  Are the same number of logger
servers
>>>>>> >>> running as tablet servers?
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> You can scan the metadata table to look for tablets
that have
>>>>>> alot of
>>>>>> >>> write-ahead logs. I think the command below will show
you how many
>>>>>> >>> write-ahead logs each tablet has.  Look for any tablets
that have
>>>>>> too
>>>>>> >>> many. I think it should sort the tablets with the most
tablets to
>>>>>> the
>>>>>> >>> top, but not positive.
>>>>>> >>>
>>>>>> >>>   ./bin/accumulo shell -u root -p secret -e 'scan -t
!METADATA -c
>>>>>> >>> log' | cut -f 1 -d ' ' | uniq -c  | sort -r -n
>>>>>> >>>
>>>>>> >>> I think the following command will show you how many
active log
>>>>>> each
>>>>>> >>> logger has.  This should be even.
>>>>>> >>>
>>>>>> >>>   ./bin/accumulo shell -u root -p secret -e 'scan -t
!METADATA -c
>>>>>> >>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 | sort | uniq
-c
>>>>>> >>>
>>>>>> >>> You can use the "flush -p" command in the shell to force
data in
>>>>>> >>> memory to disk and stop referencing write-ahead logs.
 Maybe
>>>>>> execute
>>>>>> >>> the commands above before and after flushing.
>>>>>> >>>
>>>>>> >>> Keith
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> Thanks for the replies.  I read about the flush command
in the
>>>>>> docs but
>>>>>> >> didn't make a connection between "memory" to the write ahead
>>>>>> logs.  Is
>>>>>> >> that correct?  Flush writes write ahead log data to hdfs?
>>>>>> >>
>>>>>> >> Thanks!
>>>>>> >>
>>>>>> >> -Kris
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message