hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bharath Ravi <bharathra...@gmail.com>
Subject Re: Improving HBase read performance (based on YCSB)
Date Tue, 14 Feb 2012 23:06:51 GMT
Thanks Todd!

I check disk bandwidth by first running "hparm" on it, (this shows me a
read b/w of around 56Mbps)
and then running "iftop" while the benchmarks run (This shows me that reads
are only around 10-15Mbps: but
this could definitely be because random seeks are a bottleneck)

The iostat output seems to suggest seek is a problem too, although I'm not
sure I interpret these numbers correctly.
Here's some output from iostat, while the benchmark runs:

Do the queue-lengths I see here indicate a bottleneck?

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               1.00     1.28    6.31    2.57     0.43     1.01   332.30
    3.63  408.90  259.25  776.48   6.40   5.67
dm-0              0.00     0.00    6.55    2.87     0.43     1.01   311.48
    4.60  487.90  380.14  733.49   5.99   5.65
dm-1              0.00     0.00    0.29    0.88     0.00     0.00     8.00
    1.33 1135.17   89.17 1479.15   3.23   0.38

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               1.40    18.20  257.80    2.60    13.84     0.08   109.49
   62.55  240.45  241.44  141.85   4.43 115.28
dm-0              0.00     0.00  258.80    3.40    13.81     0.01   107.99
   63.17  241.17  241.93  183.76   4.40 115.28
dm-1              0.00     0.00    0.00   17.20     0.00     0.07     8.00
    0.21   12.00    0.00   12.00   0.14   0.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               1.60   255.80  234.60    3.00    16.43     1.01   150.36
  113.11  427.74  430.83  186.40   4.82 114.56
dm-0              0.00     0.00  262.00    1.00    18.03     0.00   140.44
  113.86  389.06  389.87  175.20   4.36 114.56
dm-1              0.00     0.00    0.20  258.00     0.00     1.01     8.00
   37.92  146.87    0.00  146.98   1.02  26.32

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               3.42    98.67  152.09    2.66     9.47     0.40   130.68
   75.82  543.73  549.56  210.86   4.61  71.33
dm-0              0.00     0.00  132.32    1.33     8.04     0.01   123.43
   76.06  631.83  635.09  308.00   5.34  71.33
dm-1              0.00     0.00    3.04   99.62     0.01     0.39     8.00
   14.84  144.57  648.75  129.18   2.72  27.91

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               1.27    54.85  220.89    2.95    13.15     0.23   122.44
   66.15  304.77  305.75  231.71   4.43  99.16
dm-0              0.00     0.00  232.49    3.38    14.07     0.02   122.30
   66.66  291.36  292.25  230.00   4.20  99.16
dm-1              0.00     0.00    0.00   54.22     0.00     0.21     8.00
   18.12  334.27    0.00  334.27   1.57   8.52

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               4.00    19.80  224.60    3.00    12.80     0.09   115.95
   62.69  254.17  256.84   54.67   4.39  99.92
dm-0              0.00     0.00  229.40    2.20    13.00     0.01   115.01
   61.95  246.34  247.99   73.82   4.29  99.28
dm-1              0.00     0.00    8.00   20.40     0.03     0.08     8.00
    3.78  133.13  216.40  100.47  14.25  40.48

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               0.59   467.12  191.98    5.28    11.99     1.84   143.64
   64.63  355.47  359.03  226.22   5.02  99.10
dm-0              0.00     0.00  174.76    2.54    10.82     0.01   125.05
   64.04  392.46  396.73   99.38   5.59  99.10
dm-1              0.00     0.00    0.00  469.67     0.00     1.83     8.00
  129.15  274.97    0.00  274.97   0.26  12.05

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               4.91     0.41  228.02    0.82    12.08     0.00   108.15
   59.24  241.96  242.07  210.00   4.37  99.96
dm-0              0.00     0.00  218.00    1.02    12.97     0.00   121.31
   52.74  223.31  223.21  244.00   4.56  99.96
dm-1              0.00     0.00   28.43    0.00     0.11     0.00     8.00
    9.19  299.22  299.22    0.00  25.76  73.21

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               5.20     0.60  233.40    1.20    13.55     0.01   118.34
   81.72  316.28  317.00  177.33   4.26 100.00
dm-0              0.00     0.00  243.80    1.00    14.17     0.01   118.60
   81.06  302.02  302.52  180.80   4.08  99.92
dm-1              0.00     0.00    9.60    0.80     0.04     0.00     8.00
    8.72  496.15  522.75  177.00  96.15 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               5.60     0.80  205.20    1.80    13.77     0.01   136.35
  120.57  559.97  562.99  215.11   4.83 100.00
dm-0              0.00     0.00  203.60    1.80    13.86     0.01   138.27
  120.66  565.62  568.50  239.56   4.87 100.00
dm-1              0.00     0.00    4.80    0.60     0.02     0.00     8.00
   12.45 2189.93 2434.83  230.67 136.44  73.68

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda              12.20     0.20  208.00    1.60    13.31     0.01   130.18
  102.09  540.08  542.48  228.00   4.77 100.00
dm-0              0.00     0.00  196.40    1.40    13.09     0.01   135.61
  100.83  563.25  565.48  249.71   5.02  99.36
dm-1              0.00     0.00   25.00    0.20     0.10     0.00     8.00
   15.92  790.03  794.46  236.00  39.68 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               8.40     0.60  210.20    2.60    12.13     0.02   116.87
  101.56  443.61  445.35  302.77   4.70 100.00
dm-0              0.00     0.00  189.40    2.20    12.12     0.01   129.66
  104.66  509.39  511.55  324.00   5.22 100.00
dm-1              0.00     0.00   27.80    0.80     0.11     0.00     8.00
   11.54  386.55  391.17  226.00  34.97 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               7.73    73.54  206.87    2.06    13.89     0.26   138.64
  113.54  587.97  591.64  219.33   4.79 100.07
dm-0              0.00     0.00  190.21    1.03    12.46     0.00   133.43
  111.91  633.26  634.96  319.33   5.23 100.07
dm-1              0.00     0.00    7.22   74.57     0.03     0.29     8.00
   24.64  295.01 2199.43  110.71  12.24 100.07

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda              22.01   207.89  233.49   14.35    10.57     0.92    94.90
   63.80  245.64  258.79   31.80   4.03  99.90
dm-0              0.00     0.00  162.44    0.00    10.66     0.00   134.42
   54.57  328.21  328.21    0.00   6.09  98.95
dm-1              0.00     0.00  100.00  222.01     0.39     0.87     8.00
   24.66   77.74  113.21   61.76   3.10  99.90

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda               8.00     0.80  185.40    1.00    12.21     0.01   134.20
   36.76  195.85  196.16  136.80   5.36 100.00
dm-0              0.00     0.00  180.40    1.20    12.10     0.00   136.56
   32.82  182.95  183.42  112.00   5.51 100.00
dm-1              0.00     0.00    9.60    0.40     0.04     0.00     8.00
   17.82  420.32  428.67  220.00  99.60  99.60

On 13 February 2012 23:43, Bharath Ravi <bharathravi1@gmail.com> wrote:

> Hi all,
>
> I have a distributed HBase setup, on which I'm running the YCSB<https://github.com/brianfrankcooper/YCSB/wiki/running-a-workload>benchmark.
> There are 5 region servers, each a Dual core with around 4GB of memory,
> connected simply by a 1Gbps ethernet switch.
>
> The number of "handlers" per regionserver is set to 500 (!) and HDFS's
> maximum receivers per datanode is 4096.
>
>  The benchmark dataset is large enough not to fit in memory.
> Update/Insert/Write throughput goes up to 8000 ops/sec easily.
> However, I see read latencies in the order of seconds, and read
> throughputs of only a few 100 ops per second.
>
> "Top" tells me that the CPU's on regionservers spend 70-80% of their time
> waiting for IO, while disk and network
> have plenty of unused bandwidth. How could I diagnose where the read
> bottleneck is?
>
> Any help would be greatly appreciated :)
>
> Thanks in advance!
> --
> Bharath Ravi
>
>


-- 
Bharath Ravi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message