hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars George <lars.geo...@gmail.com>
Subject Re: [jira] Commented: (HBASE-2021) Add compaction details to master UI
Date Tue, 05 Jan 2010 21:14:07 GMT
Hi,

I tested this now on a larger cluster and the initial wait will
definitely cause frustration. I will go with plan be and load the
fragmentation asynchronously. Any preferred way do do that? Small
thread class like the other workers or is there a better way? Is that
OK? One more thread in the master which is idle most of the time
compared to the RS's.

Lars

On Tue, Jan 5, 2010 at 4:14 PM, Lars George (JIRA) <jira@apache.org> wrote:
>
>    [ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796693#action_12796693
]
>
> Lars George commented on HBASE-2021:
> ------------------------------------
>
> I added those "title" attributes to the headers with some "soothing" explanation. Do
you think that is not good enough?
>
>> Add compaction details to master UI
>> -----------------------------------
>>
>>                 Key: HBASE-2021
>>                 URL: https://issues.apache.org/jira/browse/HBASE-2021
>>             Project: Hadoop HBase
>>          Issue Type: Improvement
>>            Reporter: Lars George
>>            Assignee: Lars George
>>            Priority: Minor
>>             Fix For: 0.20.3, 0.21.0
>>
>>         Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch,
HBASE-2021.patch
>>
>>
>> There are two issues with this, first to detect that there is a compaction needed.
You can currently use the little helper util that checks if a table has at least one colfam
with more than one store file. I though about scanning all tables and all colfams in each
and then compute the "fragmentation" ratio as a percentage of colfams with more than one store
to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor
percentage are normal under insert operations it is still important to know how bad the fragmentation
is overall.
>> Another idea is to weigh the number of files per store too, so that if you have two
per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high".
Not sure how that can be done yet but noting the idea down here.
>> Of course seeing the .META. fragmentation is useful to quickly debug performance
issues (as JD told me on IRC).
>> The other issue is that when you have started a compaction you have no idea how far
it is and if it is still in progress. One indication of course is the above value. If it is
0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple
status that compactions are still in progress.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

Mime
View raw message