hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Schubert Zhang <zson...@gmail.com>
Subject Re: HBase-0.20.0 Performance Evaluation
Date Tue, 18 Aug 2009 18:27:13 GMT
My GC oprions:  -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
Yes, it is >8ms now for random read.

In my previous report, the random-read evaluation was started soon after of
the sequential-write(from empty) evaluation. The result of such case is
still good.

We have Ganglia, but I cannot access it since I am remote accessing my
cluster.  And the swap vm.swappiness=20 to minimize swap.

I just check more things carefully by commands (top, free, iostat, etc.),
and find following issue:

On my slave node (1-4) which has 4CPU cores and 8GB RAM, 2 SATA. The memory
and regionserver heap are both adequate. CPU is not very busy. Seems every
thing is ok.

 PID  USER      PR  NI  VIRT    RES  SHR S  %CPU %MEM    TIME+   COMMAND
 4398 schubert  23   0    2488m 2.0g  9.9m S   38      26.2        59:06.21
 java (it is region server)


But I am surprising that the node(5) which has 8CPU cores and 4GB RAM, 6
SATA-RAID1, has problem.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               7.46     0.00     3.28       23.11      0.00       66.15
Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda              84.83    25.12 485.57  2.49 53649.75   220.90   110.38
9.20   18.85   2.04  99.53
dm-0              0.00     0.00  0.00 25.12     0.00   201.00     8.00
0.01    0.27   0.01   0.02
dm-1              0.00     0.00 570.90  2.49 53655.72    19.90    93.61
10.74   18.72   1.74  99.53

It seems the disk I/O is very busy.
And top:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
21037 schubert  20   0 2628m 1.6g  10m S   48     41.3          71:24.74
java

I will check more when I go to company in the moning.
On Wed, Aug 19, 2009 at 1:36 AM, stack <stack@duboce.net> wrote:

> What do you have for GC config Schubert?  Now its 8ms a random read?
> St.Ack
>
> On Tue, Aug 18, 2009 at 10:28 AM, Jonathan Gray <jlist@streamy.com> wrote:
>
> > Schubert,
> >
> > I can't think of any reason your random reads would get slower after
> > inserting more data, besides GC issues.
> >
> > Do you have GC logging and JVM metrics logging turned on?  I would
> inspect
> > those to see if you have any long-running GC pauses, or just lots and
> lots
> > of GC going on.
> >
> > If I recall, you are running on 4GB nodes, 2GB RS heap, and cohosted
> > DataNodes and TaskTrackers.  We ran for a long time on a similar setup,
> but
> > once we moved to 0.20 (and to the CMS garbage collector), we really
> needed
> > to add more memory to the nodes and increase RS heap to 4 or 5GB.  The
> CMS
> > GC is less efficient in memory, but if given sufficient resources, is
> much
> > better for overall performance/throughput.
> >
> > Also, do you have Ganglia setup?  Are you seeing swapping on your RS
> nodes?
> >  Is there high IO-wait CPU usage?
> >
> > JG
> >
> >
> > Schubert Zhang wrote:
> >
> >> Addition.
> >> Only random-reads become very slow, scans and sequential-reads are ok.
> >>
> >>
> >> On Tue, Aug 18, 2009 at 6:02 PM, Schubert Zhang <zsongbo@gmail.com>
> >> wrote:
> >>
> >>  stack and J-G, Thank you very much for your helpful comment.
> >>>
> >>> But now, we find such a critical issue for random reads.
> >>> I use sequentical-writes to insert 5GB of data in our HBase table from
> >>> empty, and ~30 regions are generated. Then the random-reads takes about
> >>> 30
> >>> minutes to complete. And then, I run the sequentical-writes again.
> Thus,
> >>> another version of each cell are inserted, thus ~60 regions are
> >>> generated.
> >>> But, we I ran the random-reads again to this table, it always take long
> >>> time
> >>> (more than 2 hours).
> >>>
> >>> I check the heap usage and other metrics, does not find the reason.
> >>>
> >>> Bellow is the status of one region server:
> >>> request=0.0, regions=13, stores=13, storefiles=14,
> storefileIndexSize=2,
> >>> memstoreSize=0, usedHeap=1126, maxHeap=1991, blockCacheSize=338001080,
> >>> blockCacheFree=79686056, blockCacheCount=5014, blockCacheHitRatio=55
> >>>
> >>> Schubert
> >>>
> >>>
> >>> On Tue, Aug 18, 2009 at 5:02 AM, Schubert Zhang <zsongbo@gmail.com>
> >>> wrote:
> >>>
> >>>  We have just done a Performance Evaluation on HBase-0.20.0.
> >>>> Refers to:
> >>>>
> >>>>
> http://docloud.blogspot.com/2009/08/hbase-0200-performance-evaluation.html
> >>>>
> >>>>
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message