Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGyfEV_yRCFzAzs5eOG3c7RYpRY=aqPS5_ybcQP_K-Hp2S412w@mail.gmail.com>
References: <CAGyfEV9Whivgun3R4Wa9YDUR_v1UZ-2w7oPJR=Zrp6NnxpHjwA@mail.gmail.com>
 <CALte62zcYjbdXQcHEH8QSoSe=31uV9AAUcPPu2+nGywZxu=q5g@mail.gmail.com>
 <CAGyfEV_W-Sm8p---7Nqmk9GLX0h40GHOxi0Bdm=h18cNJ=KPTw@mail.gmail.com>
 <CALte62wrV3rdDyPNAXKQv+8b2YZ2KvvCtBKYj1Gmf4nYVB--gQ@mail.gmail.com> <CAGyfEV_yRCFzAzs5eOG3c7RYpRY=aqPS5_ybcQP_K-Hp2S412w@mail.gmail.com>
From: Nick Dimiduk <ndimiduk@gmail.com>
Date: Fri, 18 Nov 2016 17:18:40 -0800
Message-ID: <CANZa=GtQnDdyNB--LdqyiLXO5+00RTTRijq9-yPEVYOFkjSkLg@mail.gmail.com>
Subject: Re: Why hbase doesn't remove the empty region
To: hbase-dev <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7bb70e2e3164d005419d35c4
archived-at: Sat, 19 Nov 2016 01:19:13 -0000

--047d7bb70e2e3164d005419d35c4
Content-Type: text/plain; charset=UTF-8

HBase will not remove empty regions. It assumes you know what you're doing.

In 1.2 there's a new "Region Normalizer" [0] feature that runs in the
Master, which acts a bit like the Balancer, but for region sizes. I think
it's still considered experimental, so YMMV. We're still on 1.1, so I
haven't tried it in prod yet.

I also have this problem in our cluster, because of a combination of TTL
and Phoenix's Rowkey Timestamp feature. This results in region boundaries
including the timestamp, and data that goes away after a period. Eventually
all regions will become empty. I have a script that runs periodically to
calculate region sizes and drop empty regions. I volunteered to contribute
it [1], but haven't pulled it out of our puppet infra in the form of a
patch. Let me revive that thread internally.

-n

[0]: https://issues.apache.org/jira/browse/HBASE-13103
[1]: https://issues.apache.org/jira/browse/HBASE-15712

On Fri, Nov 18, 2016 at 4:59 PM, Xi Yang <alex.xi.yang@gmail.com> wrote:

> Yes, we're still on old release of 0.98.x
>
> Many of empty region didn't be deleted. Take one for example:
>
>
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_storeCount"
> > : 1,
> >
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_storeFileCount"
> > : 1,
> >
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_memStoreSize"
> > : 424,
> >
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_storeFileSize"
> > : 549,
> >
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_compactionsCompletedCount"
> > : 4,
> >
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_numBytesCompactedCount"
> > : 8616506268,
> >
> > "Namespace_default_table_java_app_logs_region_
> 29d4fa5482d17daf813e12732155e0ee_metric_numFilesCompactedCount"
> > : 4,
>
>
> The log about last time major compaction of this region is:
>
> 2016-11-03 22:45:33,453 INFO org.apache.hadoop.hbase.regionserver.HStore:
> > Completed major compaction of 1 (all) file(s) in log of
> > java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375-
> 178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.
> 29d4fa5482d17daf813e12732155e0ee.
> > into 47b8ee796fc74ff0b82222bf60318f99(size=549), total size for store is
> > 549. This selection was in queue for 0sec, and took 0sec to execute.
> > 2016-11-03 22:45:33,454 INFO
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> > compaction: Request =
> > regionName=java_app_logs,CM-sjcmhpcapp01-1476076988799-
> 135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.
> 29d4fa5482d17daf813e12732155e0ee.,
> > storeName=log, fileCount=1, fileSize=1.3 G, priority=9,
> > time=25652433393376299; duration=0sec
>
>
> You can find that its size is only 549 bytes at that time. But HBase didn't
> remove it.
>
> Here is the result I run HFile tool
>
> $ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f
> > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/
> 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99
> > 16/11/18 16:48:25 INFO Configuration.deprecation: hadoop.native.lib is
> > deprecated. Instead, use io.native.lib.available
> > 16/11/18 16:48:29 INFO Configuration.deprecation: fs.default.name is
> > deprecated. Instead, use fs.defaultFS
> > Scanning ->
> > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/
> 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99
> > 16/11/18 16:48:29 INFO hfile.CacheConfig: CacheConfig:disabled
> > Scanned kv count -> 0
>
>
>
> 2016-11-17 18:45 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>
> > You would be able to find in region server log(s).
> >
> > Can you do quick inspection of the hfiles using:
> >
> > http://hbase.apache.org/book.html#hfile_tool
> >
> > BTW you're still on old release of 0.98.x, right ?
> >
> > On Thu, Nov 17, 2016 at 6:40 PM, Xi Yang <alex.xi.yang@gmail.com> wrote:
> >
> > > Thank you for reply.
> > >
> > > Yes. There are many regions have this problem. And those regions were
> > > created several years ago.
> > > hbase.hregion.majorcompaction of our cluster is 7 days.
> > > Where can I see the last major compaction time?
> > >
> > > Thanks,
> > > Alex
> > >
> > > 2016-11-17 18:22 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
> > >
> > > > When was the last time major compaction was performed on this region
> ?
> > > >
> > > > Were you referring to the store files by 'them' in your question ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Nov 17, 2016 at 6:18 PM, Xi Yang <alex.xi.yang@gmail.com>
> > wrote:
> > > >
> > > > > I have some regions are empty. But HBase doesn't remove them, why?
> > > > >
> > > > > You can see the storeFileSize is only 549. But I use hbase shell to
> > > scan
> > > > > this region, the result is empty
> > > > >
> > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileCount"
> > > > > : 1,
> > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_memStoreSize"
> > > > > : 424,
> > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileSize"
> > > > > : 549,
> > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_compactionsCompletedCount"
> > > > > : 4,
> > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numBytesCompactedCount"
> > > > > : 8503271111,
> > > > >     "Namespace_default_table_java_app_logs_region_
> > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numFilesCompactedCount"
> > > > > : 4,
> > > > >
> > > >
> > >
> >
>

--047d7bb70e2e3164d005419d35c4--