hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: write performance degradation
Date Sun, 30 Jan 2011 22:49:32 GMT
RC3 was promoted to 0.90 release, so I didn't expect much difference.
I also tried 0.90 build which also included newer patches such as HBASE-3455
and observed same result.

The cluster only runs one job at a time.

Client buffer size for our 0.20.6 cluster was set to 20MB and I didn't see
similar issue.

Here is output from top on one node:

top - 22:42:26 up 104 days, 20:38,  1 user,  load average: 1.04, 1.34, 2.13
Tasks: 152 total,   1 running, 151 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us,  0.2%sy,  0.0%ni, 98.8%id,  0.0%wa,  0.0%hi,  0.4%si,
0.0%st
Mem:  16440036k total, 15532616k used,   907420k free,    42472k buffers
Swap: 16777208k total,  5719488k used, 11057720k free,  4530368k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND

15261 hadoop    25   0 4776m 1.0g 9312 S  2.0  6.7 133:27.04 java
16442 hadoop    21   0 1625m 760m  14m S  1.3  4.7   0:39.28
java

16453 hadoop    18   0 4690m 3.4g 9712 S  1.3 21.4 311:49.25 java

15261 DataNode
19442 map task
16453 HRegionServer

---------
[sjc1-hadoop3.sjc1:hadoop 1298]iostat
Linux 2.6.18-53.el5 (sjc1-hadoop3.sjc1.carrieriq.com)   01/30/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.56    0.00    0.48    0.60    0.00   94.35

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               6.15       403.62       411.58 3656841492 3728977064
sdb               7.64       131.10       303.53 1187817806 2750029496
sdc               7.38       396.23       387.12 3589886886 3507388096
sdd               6.35       403.99       399.77 3660260454 3621972144
dm-0             22.51        37.78       173.52  342296914 1572156872
dm-1             27.92        93.32       130.01  845519592 1177872624

On Sun, Jan 30, 2011 at 2:34 PM, Stack <stack@duboce.net> wrote:

> On Sun, Jan 30, 2011 at 3:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > Using 0.90 RC3 on our 7 node clsuter, the first time (after cluster
> start)
> > our map tasks ran quite fast writing to hbase (client side buffer size
> being
> > 17MB).
> >
>
> Can you come up on release 0.90.0 Ted?
>
> 17MB client buffer might be a little large.
>
>
> > If I run the same hadoop job soon after the first finishes, there is
> > noticeable slow down.
> >
>
> What do you see in your ganglia?  Anything else running on cluster at
> the time?  What are the machines doing?  CPU, iowait?  Swapping?
>
> Thanks,
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message