Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B52E49727 for ; Thu, 12 Apr 2012 15:11:14 +0000 (UTC) Received: (qmail 52909 invoked by uid 500); 12 Apr 2012 15:11:14 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 52889 invoked by uid 500); 12 Apr 2012 15:11:14 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 52881 invoked by uid 99); 12 Apr 2012 15:11:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2012 15:11:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of eric.newton@gmail.com designates 209.85.210.169 as permitted sender) Received: from [209.85.210.169] (HELO mail-iy0-f169.google.com) (209.85.210.169) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2012 15:11:07 +0000 Received: by iajr24 with SMTP id r24so3600412iaj.0 for ; Thu, 12 Apr 2012 08:10:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=5bKN7i6COQ4SAdu9/TGhSvZU+seOKEgIVtNHICZJcYo=; b=j7mlOoPY4+c5ZNJ64gckLJg/UxdCQrHVqzpMG1hysmMaRSijhKV23jVm9ODTaAQuo2 ldbWJoRe5E1EXHS/K2ARfWJXUd6JfkV5CJ9K9tnkNV9pOlzMQCSS1Y7KqwbLYq4RO6LB jMBWXSpR8tTme49qNEBh3FR1f18uRgfV7vGzvzwOcuZjHNJoglC84MgDLtOtsDls0lq+ nN1ugXFpfnxgAUEqX1kaKy3XysmePtcXXJzoGmZCLow5iyWciNxiqjkKAJ3h5ixnpKsl t5NQiGlDUj1GI1apP0T3+tZQ08D9gQLt2RoHGosgxDO/p6WHi1sOiyA7+y1DrBO+YMdh pt/g== MIME-Version: 1.0 Received: by 10.50.155.229 with SMTP id vz5mr2847442igb.12.1334242977012; Thu, 12 Apr 2012 08:02:57 -0700 (PDT) Received: by 10.50.158.196 with HTTP; Thu, 12 Apr 2012 08:02:56 -0700 (PDT) In-Reply-To: References: Date: Thu, 12 Apr 2012 11:02:56 -0400 Message-ID: Subject: Re: Write ahead log growth and reduction From: Eric Newton To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=e89a8f3baa67b05cf004bd7ca6d0 --e89a8f3baa67b05cf004bd7ca6d0 Content-Type: text/plain; charset=ISO-8859-1 Running the tracer is optional. Just make an empty tracers file in the conf directory and restart. -Eric On Thu, Apr 12, 2012 at 10:21 AM, Kristopher Kane wrote: > Eric, thanks for the help. > > This is on 1.3.2, looks like we will be able to go to 1.3.4 very shortly. > From the below JIRA, it looks like it is fixed in 1.3.6. I will send this > up as I'm not an approving authority on installed/running versions and will > also recommend the trace password in the site.xml. > > Once we upgrade to the minor version, I will report back on whether the > WAL GC runs in that version. > > -Kris > > On Wed, Apr 11, 2012 at 1:00 PM, Eric Newton wrote: > >> Also you will need to restart. >> >> What version of accumulo are you running? This problem should have been >> fixed with ACCUMULO-243. >> >> -Eric >> >> On Wed, Apr 11, 2012 at 12:28 PM, Eric Newton wrote: >> >>> in conf/accumulo-site.xml, add this: >>> >>> >>> trace.password >>> your-root-password-here >>> >>> >>> -Eric >>> >>> On Wed, Apr 11, 2012 at 12:14 PM, Kristopher Kane wrote: >>> >>>> I know I haven't answered some of the previous questions to me, but, >>>> the GC talk brought me to the source of the problem. our WAL GC job is not >>>> successfully running. The GC logs state a bad username/password, so, will >>>> investigate that. >>>> >>>> Thanks for the help! >>>> >>>> -Kris >>>> >>>> On Wed, Apr 11, 2012 at 11:18 AM, Keith Turner wrote: >>>> >>>>> When data is written to the accumulo its written to memory and the >>>>> write ahead logs. The data in memory is sorted and the data in the >>>>> write ahead logs is written asis (unsorted). When the data in memory >>>>> is flushed to HDFS, the write ahead logs that also contain that data >>>>> are no longer needed. >>>>> >>>>> On Wed, Apr 11, 2012 at 11:12 AM, Eric Newton >>>>> wrote: >>>>> > Logs are only put into HDFS during a recovery. >>>>> > >>>>> > Flush removes references to WALs, and the accumulo gc will ask the >>>>> loggers >>>>> > to delete them when there are no references to them. >>>>> > >>>>> > -Eric >>>>> > >>>>> > >>>>> > On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane < >>>>> kkane.list@gmail.com> >>>>> > wrote: >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner >>>>> wrote: >>>>> >>> >>>>> >>> How big is the partition? Are the same number of logger servers >>>>> >>> running as tablet servers? >>>>> >>> >>>>> >>> >>>>> >>> You can scan the metadata table to look for tablets that have alot >>>>> of >>>>> >>> write-ahead logs. I think the command below will show you how many >>>>> >>> write-ahead logs each tablet has. Look for any tablets that have >>>>> too >>>>> >>> many. I think it should sort the tablets with the most tablets to >>>>> the >>>>> >>> top, but not positive. >>>>> >>> >>>>> >>> ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c >>>>> >>> log' | cut -f 1 -d ' ' | uniq -c | sort -r -n >>>>> >>> >>>>> >>> I think the following command will show you how many active log >>>>> each >>>>> >>> logger has. This should be even. >>>>> >>> >>>>> >>> ./bin/accumulo shell -u root -p secret -e 'scan -t !METADATA -c >>>>> >>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 | sort | uniq -c >>>>> >>> >>>>> >>> You can use the "flush -p" command in the shell to force data in >>>>> >>> memory to disk and stop referencing write-ahead logs. Maybe >>>>> execute >>>>> >>> the commands above before and after flushing. >>>>> >>> >>>>> >>> Keith >>>>> >>> >>>>> >>> >>>>> >> >>>>> >> >>>>> >> >>>>> >> Thanks for the replies. I read about the flush command in the docs >>>>> but >>>>> >> didn't make a connection between "memory" to the write ahead logs. >>>>> Is >>>>> >> that correct? Flush writes write ahead log data to hdfs? >>>>> >> >>>>> >> Thanks! >>>>> >> >>>>> >> -Kris >>>>> > >>>>> > >>>>> >>>> >>>> >>> >> > --e89a8f3baa67b05cf004bd7ca6d0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Running the tracer is optional. =A0Just make an empty tracers file in the c= onf directory and restart.

-Eric

On Thu, Apr 12, 2012 at 10:21 AM, Kristopher Kane <kkane.list@gmai= l.com> wrote:
Eric, thanks for the help.
= =A0
This is on 1.3.2, looks like we will be able to go to 1.3.4 v= ery shortly.=A0 From the below JIRA, it looks like it is fixed in 1.3.6.=A0= I will send this up as I'm not an approving authority on installed/run= ning versions and will also recommend the trace password in the site.xml. <= /div>
=A0
Once we upgrade to the minor version, I will report back= on whether the WAL GC runs in that version.
=A0
-Kris=

On Wed, Apr 11, 2012 at 1:00 PM, Eric Newton <eric.newton@gmail.com> wrote:
Also you will need to restart.

What ver= sion of accumulo are you running? =A0This problem should have been fixed wi= th=A0ACCUMULO-243.

-Eric=

On Wed, Apr 11, 2012 at 12:28 PM, Eric Newton <eric.newton@gmail.com> wrote:
in conf/accumulo-site.xml, add this:

=A0 =A0 <property>
=A0 =A0 =A0 <name>trace.password&l= t;/name>
=A0 =A0 =A0 <value>your-root-password-here</value>
=A0 =A0 </property>

-Eric

On Wed, Apr 11, 2012 at 12:14 PM, Kristopher Kane <kkane.lis= t@gmail.com> wrote:
I know I haven't answered some of the previous questions to me, bu= t, the GC talk brought me to the source of the problem.=A0 our WAL GC job i= s not successfully running.=A0 The GC logs state a bad username/password, s= o, will investigate that.
=A0
Thanks for the help!
=A0
-Kris
On Wed, Apr 11, 2012 at 11:18 = AM, Keith Turner <keith@deenlo.com> wrote:
When data is written to the accumulo its written to memory= and the
write ahead logs. =A0The data in memory is sorted and the data in the
write ahead logs is written asis (unsorted). =A0When the data in memory
is flushed to HDFS, the write ahead logs that also contain that data
are no longer needed.

On Wed, Apr 11, 2012 at 11:12 AM, Eric Newton <eric.newton@gmail.com> wrote:
> Logs are only put into HDFS during a recovery.
>
> Flush removes references to WALs, and the accumulo gc will ask the log= gers
> to delete them when there are no references to them.
>
> -Eric
>
>
> On Wed, Apr 11, 2012 at 11:02 AM, Kristopher Kane <kkane.list@gmail.com>
> wrote:
>>
>>
>>
>> On Wed, Apr 11, 2012 at 10:48 AM, Keith Turner <keith@deenlo.com> wrote:
>>>
>>> How big is the partition? =A0Are the same number of logger ser= vers
>>> running as tablet servers?
>>>
>>>
>>> You can scan the metadata table to look for tablets that have = alot of
>>> write-ahead logs. I think the command below will show you how = many
>>> write-ahead logs each tablet has. =A0Look for any tablets that= have too
>>> many. I think it should sort the tablets with the most tablets= to the
>>> top, but not positive.
>>>
>>> =A0 ./bin/accumulo shell -u root -p secret -e 'scan -t !ME= TADATA -c
>>> log' | cut -f 1 -d ' ' | uniq -c =A0| sort -r -n >>>
>>> I think the following command will show you how many active lo= g each
>>> logger has. =A0This should be even.
>>>
>>> =A0 ./bin/accumulo shell -u root -p secret -e 'scan -t !ME= TADATA -c
>>> log' | cut -f 2 -d ' ' | cut -d ':' -f 2 |= sort | uniq -c
>>>
>>> You can use the "flush -p" command in the shell to f= orce data in
>>> memory to disk and stop referencing write-ahead logs. =A0Maybe= execute
>>> the commands above before and after flushing.
>>>
>>> Keith
>>>
>>>
>>
>>
>>
>> Thanks for=A0the replies.=A0 I read=A0about the flush command in t= he docs but
>> didn't make a connection between "memory" to the wri= te ahead logs.=A0 Is
>> that=A0correct?=A0 Flush writes write ahead=A0log data to hdfs? >>
>> Thanks!
>>
>> -Kris
>
>





--e89a8f3baa67b05cf004bd7ca6d0--