Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of lars.george@gmail.com
 designates 209.85.219.214 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KH6/VmaboxojTfGyXwIrdhreMWk1hD2mq2vPnZEq+a6adF2miySd/S7H52MwrskLHc
         TcYfptZRfkK+BZsuM7JHvMIoGRztR65nI7PF60ZWfOXR5CzXxT50E3HIupfdSfmY0xnh
         vji1KBIyzbu1WGK3nyc6vEOCD5CGdwdgUInPQ=
MIME-Version: 1.0
In-Reply-To: <1069422434.44901262704494784.JavaMail.jira@brutus.apache.org>
References: <487332857.1259691621014.JavaMail.jira@brutus>
	 <1069422434.44901262704494784.JavaMail.jira@brutus.apache.org>
Date: Tue, 5 Jan 2010 22:14:07 +0100
Message-ID: <61770b881001051314x6dc18423l5ddcb2a80b2124e@mail.gmail.com>
Subject: Re: [jira] Commented: (HBASE-2021) Add compaction details to master
	UI
From: Lars George <lars.george@gmail.com>
To: hbase-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

I tested this now on a larger cluster and the initial wait will
definitely cause frustration. I will go with plan be and load the
fragmentation asynchronously. Any preferred way do do that? Small
thread class like the other workers or is there a better way? Is that
OK? One more thread in the master which is idle most of the time
compared to the RS's.

Lars

On Tue, Jan 5, 2010 at 4:14 PM, Lars George (JIRA) <jira@apache.org> wrote:
>
> =A0 =A0[ https://issues.apache.org/jira/browse/HBASE-2021?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=
=3D12796693#action_12796693 ]
>
> Lars George commented on HBASE-2021:
> ------------------------------------
>
> I added those "title" attributes to the headers with some "soothing" expl=
anation. Do you think that is not good enough?
>
>> Add compaction details to master UI
>> -----------------------------------
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Key: HBASE-2021
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 URL: https://issues.apache.org/jira/brow=
se/HBASE-2021
>> =A0 =A0 =A0 =A0 =A0 =A0 Project: Hadoop HBase
>> =A0 =A0 =A0 =A0 =A0Issue Type: Improvement
>> =A0 =A0 =A0 =A0 =A0 =A0Reporter: Lars George
>> =A0 =A0 =A0 =A0 =A0 =A0Assignee: Lars George
>> =A0 =A0 =A0 =A0 =A0 =A0Priority: Minor
>> =A0 =A0 =A0 =A0 =A0 =A0 Fix For: 0.20.3, 0.21.0
>>
>> =A0 =A0 =A0 =A0 Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v=
3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>>
>>
>> There are two issues with this, first to detect that there is a compacti=
on needed. You can currently use the little helper util that checks if a ta=
ble has at least one colfam with more than one store file. I though about s=
canning all tables and all colfams in each and then compute the "fragmentat=
ion" ratio as a percentage of colfams with more than one store to the total=
 number of colfams. That gives a "Table xyz is 33% fragmented" output. Whil=
e minor percentage are normal under insert operations it is still important=
 to know how bad the fragmentation is overall.
>> Another idea is to weigh the number of files per store too, so that if y=
ou have two per colfam it is considered "low" and if you have more, for exa=
mple 6-8 it is considered "high". Not sure how that can be done yet but not=
ing the idea down here.
>> Of course seeing the .META. fragmentation is useful to quickly debug per=
formance issues (as JD told me on IRC).
>> The other issue is that when you have started a compaction you have no i=
dea how far it is and if it is still in progress. One indication of course =
is the above value. If it is 0% then all is done. But if you are at say 23%=
, is it still compacting? We could have a simple status that compactions ar=
e still in progress.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>