hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars George (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-2021) Add compaction details to master UI
Date Thu, 03 Dec 2009 16:33:25 GMT

     [ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Lars George updated HBASE-2021:

    Attachment: HBASE-2021.patch

HBASE-2021.patch adds

- compaction queue size to region server metrics, can be seen on regionserver.jsp

|Metrics | request=0.0, regions=3, stores=6, storefiles=3, storefileIndexSize=0, memstoreSize=0,
compactionQueueSize=0, usedHeap=35, maxHeap=994, blockCacheSize=1713760, blockCacheFree=206913552,
blockCacheCount=1, blockCacheHitRatio=95 | RegionServer Metrics; file and heap sizes are in
megabytes |

The "compactionQueueSize" reflects the number of regions this regionserver has in the queue
to be processed and allows to "determine" if compactions are in progress from the UI. 

I would have preferred to have a global indicator in the master UI but that value cannot be
bubbled up unless I for example add it to HServerInfo - which I have been told is not a good
idea to change. Right?

- Added fragmentation details to UI

Added a total fragmentation precentage to master.jsp in top table, for example:

| Regions On FS | 3 | Number of regions on FileSystem. Rough count. |
| Fragmentation | 17% | Overall fragmentation of all tables, including .META. and -ROOT-.|
|Zookeeper Quorum | localhost:2181 | Addresses of all registered ZK servers. For more, see
zk dump.|

Then in the tables below for .META., -ROOT- and user table I added it as a column with an
abbreviated title but added a pop-up explanation when you hover over it.

> Add compaction details to master UI
> -----------------------------------
>                 Key: HBASE-2021
>                 URL: https://issues.apache.org/jira/browse/HBASE-2021
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: Lars George
>            Assignee: Lars George
>            Priority: Minor
>         Attachments: HBASE-2021.patch
> There are two issues with this, first to detect that there is a compaction needed. You
can currently use the little helper util that checks if a table has at least one colfam with
more than one store file. I though about scanning all tables and all colfams in each and then
compute the "fragmentation" ratio as a percentage of colfams with more than one store to the
total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage
are normal under insert operations it is still important to know how bad the fragmentation
is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per
colfam it is considered "low" and if you have more, for example 6-8 it is considered "high".
Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues
(as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it
is and if it is still in progress. One indication of course is the above value. If it is 0%
then all is done. But if you are at say 23%, is it still compacting? We could have a simple
status that compactions are still in progress.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message