accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Newton <eric.new...@gmail.com>
Subject Re: Write ahead log growth and reduction
Date Thu, 12 Apr 2012 15:02:56 GMT
Running the tracer is optional.  Just make an empty tracers file in the
conf directory and restart.

-Eric

On Thu, Apr 12, 2012 at 10:21 AM, Kristopher Kane <kkane.list@gmail.com>wrote:

> Eric, thanks for the help.
>
> This is on 1.3.2, looks like we will be able to go to 1.3.4 very shortly.
> From the below JIRA, it looks like it is fixed in 1.3.6.  I will send this
> up as I'm not an approving authority on installed/running versions and will
> also recommend the trace password in the site.xml.
>
> Once we upgrade to the minor version, I will report back on whether the
> WAL GC runs in that version.
>
> -Kris
>
> On Wed, Apr 11, 2012 at 1:00 PM, Eric Newton <eric.newton@gmail.com>wrote:
>
>> Also you will need to restart.
>>
>> What version of accumulo are you running?  This problem should have been
>> fixed with ACCUMULO-243.
>>
>> -Eric
>>
>> On Wed, Apr 11, 2012 at 12:28 PM, Eric Newton <eric.newton@gmail.com>wrote:
>>
>>> in conf/accumulo-site.xml, add this:
>>>
>>>     <property>
>>>       <name>trace.password</name>
>>>       <value>your-root-password-here</value>
>>>     </property>
>>>
>>> -Eric
>>>
>>> On Wed, Apr 11, 2012 at 12:14 PM, Kristopher Kane <kkane.list@gmail.com>wrote:
>>>
>>>> I know I haven't answered some of the previous questions to me, but,
>>>> the GC talk brought me to the source of the problem.  our WAL GC job is not
>>>> successfully running.  The GC logs state a bad username/password, so, will
>>>> investigate that.
>>>>
>>>> Thanks for the help!
>>>>
>>>> -Kris
>>>>
>>>> On Wed, Apr 11, 2012 at 11:18 AM, Keith Turner <keith@deenlo.com>wrote:
>>>>
>>>>> When data is written to the accumulo its written to memory and the
>>>>> write ahead logs.  The data in memory is sorted and the data in the
>>>>> write ahead logs is written asis (unsorted).  When the data in memory
>>>>> is flushed to HDFS, the write ahead logs that also contain that data
>>>>> are no longer needed.
>>>>>
>>>>> On Wed, Apr 11, 2012 at 11:12 AM, Eric Newton <eric.newton@gmail.com>
>>>>> wrote:
>>>>> > Logs are only put into HDFS during a recovery.
>>>>> >
>>>>> > Flush removes references to WALs, and the accumulo gc will ask the
>>>>> loggers
>>>>> > to delete them when there are no references to them.
>>>>> >
>>>>> > -Eric
>>>>> >
>>>>> >
>>>>> > On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane <
>>>>> kkane.list@gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner <keith@deenlo.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> How big is the partition?  Are the same number of logger
servers
>>>>> >>> running as tablet servers?
>>>>> >>>
>>>>> >>>
>>>>> >>> You can scan the metadata table to look for tablets that
have alot
>>>>> of
>>>>> >>> write-ahead logs. I think the command below will show you
how many
>>>>> >>> write-ahead logs each tablet has.  Look for any tablets
that have
>>>>> too
>>>>> >>> many. I think it should sort the tablets with the most tablets
to
>>>>> the
>>>>> >>> top, but not positive.
>>>>> >>>
>>>>> >>>   ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA
-c
>>>>> >>> log' | cut -f 1 -d ' ' | uniq -c  | sort -r -n
>>>>> >>>
>>>>> >>> I think the following command will show you how many active
log
>>>>> each
>>>>> >>> logger has.  This should be even.
>>>>> >>>
>>>>> >>>   ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA
-c
>>>>> >>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 | sort | uniq -c
>>>>> >>>
>>>>> >>> You can use the "flush -p" command in the shell to force
data in
>>>>> >>> memory to disk and stop referencing write-ahead logs.  Maybe
>>>>> execute
>>>>> >>> the commands above before and after flushing.
>>>>> >>>
>>>>> >>> Keith
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Thanks for the replies.  I read about the flush command in the
docs
>>>>> but
>>>>> >> didn't make a connection between "memory" to the write ahead
logs.
>>>>> Is
>>>>> >> that correct?  Flush writes write ahead log data to hdfs?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> -Kris
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message