hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xi Yang <alex.xi.y...@gmail.com>
Subject Re: Why hbase doesn't remove the empty region
Date Sat, 19 Nov 2016 18:54:01 GMT
Thanks Nick & Ted. Thank you for your help and your patience.

"Tool for retiring empty regions" will be an useful feature!

2016-11-18 17:18 GMT-08:00 Nick Dimiduk <ndimiduk@gmail.com>:

> HBase will not remove empty regions. It assumes you know what you're doing.
>
> In 1.2 there's a new "Region Normalizer" [0] feature that runs in the
> Master, which acts a bit like the Balancer, but for region sizes. I think
> it's still considered experimental, so YMMV. We're still on 1.1, so I
> haven't tried it in prod yet.
>
> I also have this problem in our cluster, because of a combination of TTL
> and Phoenix's Rowkey Timestamp feature. This results in region boundaries
> including the timestamp, and data that goes away after a period. Eventually
> all regions will become empty. I have a script that runs periodically to
> calculate region sizes and drop empty regions. I volunteered to contribute
> it [1], but haven't pulled it out of our puppet infra in the form of a
> patch. Let me revive that thread internally.
>
> -n
>
> [0]: https://issues.apache.org/jira/browse/HBASE-13103
> [1]: https://issues.apache.org/jira/browse/HBASE-15712
>
> On Fri, Nov 18, 2016 at 4:59 PM, Xi Yang <alex.xi.yang@gmail.com> wrote:
>
> > Yes, we're still on old release of 0.98.x
> >
> > Many of empty region didn't be deleted. Take one for example:
> >
> >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_storeCount"
> > > : 1,
> > >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileCount"
> > > : 1,
> > >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_memStoreSize"
> > > : 424,
> > >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileSize"
> > > : 549,
> > >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_compactionsCompletedCount"
> > > : 4,
> > >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_numBytesCompactedCount"
> > > : 8616506268,
> > >
> > > "Namespace_default_table_java_app_logs_region_
> > 29d4fa5482d17daf813e12732155e0ee_metric_numFilesCompactedCount"
> > > : 4,
> >
> >
> > The log about last time major compaction of this region is:
> >
> > 2016-11-03 22:45:33,453 INFO org.apache.hadoop.hbase.
> regionserver.HStore:
> > > Completed major compaction of 1 (all) file(s) in log of
> > > java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375-
> > 178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.
> > 29d4fa5482d17daf813e12732155e0ee.
> > > into 47b8ee796fc74ff0b82222bf60318f99(size=549), total size for store
> is
> > > 549. This selection was in queue for 0sec, and took 0sec to execute.
> > > 2016-11-03 22:45:33,454 INFO
> > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> > > compaction: Request =
> > > regionName=java_app_logs,CM-sjcmhpcapp01-1476076988799-
> > 135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.
> > 29d4fa5482d17daf813e12732155e0ee.,
> > > storeName=log, fileCount=1, fileSize=1.3 G, priority=9,
> > > time=25652433393376299; duration=0sec
> >
> >
> > You can find that its size is only 549 bytes at that time. But HBase
> didn't
> > remove it.
> >
> > Here is the result I run HFile tool
> >
> > $ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f
> > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/
> > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99
> > > 16/11/18 16:48:25 INFO Configuration.deprecation: hadoop.native.lib is
> > > deprecated. Instead, use io.native.lib.available
> > > 16/11/18 16:48:29 INFO Configuration.deprecation: fs.default.name is
> > > deprecated. Instead, use fs.defaultFS
> > > Scanning ->
> > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/
> > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99
> > > 16/11/18 16:48:29 INFO hfile.CacheConfig: CacheConfig:disabled
> > > Scanned kv count -> 0
> >
> >
> >
> > 2016-11-17 18:45 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
> >
> > > You would be able to find in region server log(s).
> > >
> > > Can you do quick inspection of the hfiles using:
> > >
> > > http://hbase.apache.org/book.html#hfile_tool
> > >
> > > BTW you're still on old release of 0.98.x, right ?
> > >
> > > On Thu, Nov 17, 2016 at 6:40 PM, Xi Yang <alex.xi.yang@gmail.com>
> wrote:
> > >
> > > > Thank you for reply.
> > > >
> > > > Yes. There are many regions have this problem. And those regions were
> > > > created several years ago.
> > > > hbase.hregion.majorcompaction of our cluster is 7 days.
> > > > Where can I see the last major compaction time?
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > 2016-11-17 18:22 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
> > > >
> > > > > When was the last time major compaction was performed on this
> region
> > ?
> > > > >
> > > > > Were you referring to the store files by 'them' in your question
?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, Nov 17, 2016 at 6:18 PM, Xi Yang <alex.xi.yang@gmail.com>
> > > wrote:
> > > > >
> > > > > > I have some regions are empty. But HBase doesn't remove them,
> why?
> > > > > >
> > > > > > You can see the storeFileSize is only 549. But I use hbase shell
> to
> > > > scan
> > > > > > this region, the result is empty
> > > > > >
> > > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileCount"
> > > > > > : 1,
> > > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_memStoreSize"
> > > > > > : 424,
> > > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileSize"
> > > > > > : 549,
> > > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_
> compactionsCompletedCount"
> > > > > > : 4,
> > > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numBytesCompactedCount"
> > > > > > : 8503271111,
> > > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numFilesCompactedCount"
> > > > > > : 4,
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message