hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: hbase logs show the performance of compact isnt good. any suggstion to find out the reason?
Date Mon, 27 Aug 2018 19:33:34 GMT
Hello Lu:

Any other environmental or hardware differences between dev and prod? E.g.
is the compaction the only thing running on prod? Are there adjacent
processes that might be using up i/o or network? Are the machines same in
prod as in dev?

Also, you compare one compaction on dev to another on prod. If you compare
a larger sample set, do you see the phenomenon still or is it this single
compaction only?

There is a compaction tool that you might run on both clusters to more
directly compare compaction throughputs:
http://hbase.apache.org/book.html#compaction.tool

S

On Mon, Aug 20, 2018 at 3:11 AM Lu Dillon <ludi_1981@hotmail.com> wrote:

> Hi All,
>
>
> I'm a newbie against Hbase. When I use YCSB to load data into Hbase, the
> compaction in two cluster looks different.
>
> DEV: 6 nodes. A and B are master and standby for HDFS and Hbase. C/D/E/F
> are the HDFS datanode and regionservers.
>
> PROD: 8 nodes. A and B are master and standby for HDFS and Hbase. C/D/E/F/G/H
> are the HDFS datanode and regionservers.
>
> Network is 1000mbs.
>
> Hadoop: 2.7.3
>
> Hbase: 1.2.6
>
> JDK:1.8
>
> Zookeep: install on C/D/E
>
>
>
> In DEV, I notice the compaction 4G files are about 30 seconds; however, it
> tooks about 30 seconds to process 400MB files in PROD.
>
> For example, this is logs from one node of DEV:
>
> 2018-08-10 16:30:47,535 INFO  [regionserver/node103/172.28.200.103:16020-shortCompactions-1533803656579]
> regionserver.HStore: Completed compaction of 6 file(s) in family of
> usertable_nosplit,,1533878791857.be420543a609c32475386855dee93a82. into
> abe3a5c0dbbd4708ae2523f9b4a53ad8(size=1.4 G), total size for store is 4.0
> G. This selection was in queue for 0sec, and took 18sec to execute.
>
>
> This is from PROD:
>
> 2018-08-17 23:29:28,751 INFO
>  [regionserver/YJB-HADOOP-74-23/192.168.74.23:16020-shortCompactions-1534519629607]
> regionserver.HStore: Completed compaction of 3 (all) file(s) in family of
> usertable_nosplit,,1534518587365.95595277af1c56260b85de113c9687f1. into
> 934669bf6a8a407aa6ee385439c4a7b5(size=430.8 M), total size for store is
> 430.8 M. This selection was in queue for 0sec, and took 30sec to execute.
>
>
> I thought this might be a problem of GC. So I change to use CG1 with this
> article "https://product.hubspot.com/blog/g1gc-tuning-your-hbase-cluster"
> . The performance improve a little, not to good.
>
>
> Is there any suggestion to find out what causes the problem and improve
> the performance?
>
>
> BTW: I'm not sure what kind information can help so I attach my
> hbase-site.xml and hbase-env.sh first.
>
> BTW2: to use GC1 against regionserver, I create a speraeted hbase-env.sh
> for regionserver.
>
>
> Thanks,
>
> Dillon
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message