hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bharath Ravi <bharathra...@gmail.com>
Subject Re: Improving HBase read performance (based on YCSB)
Date Wed, 15 Feb 2012 14:18:08 GMT
Thanks a lot for the help Todd!

On 14 February 2012 22:39, Todd Lipcon <todd@cloudera.com> wrote:

> Yep, definitely bound on seeks - see the 100% util, and the r/s >100.
> The bandwidth provided by random IO from a disk is going to be much
> smaller than the sequential IO you see from hdparm
>
> -Todd
>
> On Tue, Feb 14, 2012 at 3:06 PM, Bharath Ravi <bharathravi1@gmail.com>
> wrote:
> > Thanks Todd!
> >
> > I check disk bandwidth by first running "hparm" on it, (this shows me a
> > read b/w of around 56Mbps)
> > and then running "iftop" while the benchmarks run (This shows me that
> reads
> > are only around 10-15Mbps: but
> > this could definitely be because random seeks are a bottleneck)
> >
> > The iostat output seems to suggest seek is a problem too, although I'm
> not
> > sure I interpret these numbers correctly.
> > Here's some output from iostat, while the benchmark runs:
> >
> > Do the queue-lengths I see here indicate a bottleneck?
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               1.00     1.28    6.31    2.57     0.43     1.01
> 332.30
> >    3.63  408.90  259.25  776.48   6.40   5.67
> > dm-0              0.00     0.00    6.55    2.87     0.43     1.01
> 311.48
> >    4.60  487.90  380.14  733.49   5.99   5.65
> > dm-1              0.00     0.00    0.29    0.88     0.00     0.00
> 8.00
> >    1.33 1135.17   89.17 1479.15   3.23   0.38
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               1.40    18.20  257.80    2.60    13.84     0.08
> 109.49
> >   62.55  240.45  241.44  141.85   4.43 115.28
> > dm-0              0.00     0.00  258.80    3.40    13.81     0.01
> 107.99
> >   63.17  241.17  241.93  183.76   4.40 115.28
> > dm-1              0.00     0.00    0.00   17.20     0.00     0.07
> 8.00
> >    0.21   12.00    0.00   12.00   0.14   0.24
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               1.60   255.80  234.60    3.00    16.43     1.01
> 150.36
> >  113.11  427.74  430.83  186.40   4.82 114.56
> > dm-0              0.00     0.00  262.00    1.00    18.03     0.00
> 140.44
> >  113.86  389.06  389.87  175.20   4.36 114.56
> > dm-1              0.00     0.00    0.20  258.00     0.00     1.01
> 8.00
> >   37.92  146.87    0.00  146.98   1.02  26.32
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               3.42    98.67  152.09    2.66     9.47     0.40
> 130.68
> >   75.82  543.73  549.56  210.86   4.61  71.33
> > dm-0              0.00     0.00  132.32    1.33     8.04     0.01
> 123.43
> >   76.06  631.83  635.09  308.00   5.34  71.33
> > dm-1              0.00     0.00    3.04   99.62     0.01     0.39
> 8.00
> >   14.84  144.57  648.75  129.18   2.72  27.91
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               1.27    54.85  220.89    2.95    13.15     0.23
> 122.44
> >   66.15  304.77  305.75  231.71   4.43  99.16
> > dm-0              0.00     0.00  232.49    3.38    14.07     0.02
> 122.30
> >   66.66  291.36  292.25  230.00   4.20  99.16
> > dm-1              0.00     0.00    0.00   54.22     0.00     0.21
> 8.00
> >   18.12  334.27    0.00  334.27   1.57   8.52
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               4.00    19.80  224.60    3.00    12.80     0.09
> 115.95
> >   62.69  254.17  256.84   54.67   4.39  99.92
> > dm-0              0.00     0.00  229.40    2.20    13.00     0.01
> 115.01
> >   61.95  246.34  247.99   73.82   4.29  99.28
> > dm-1              0.00     0.00    8.00   20.40     0.03     0.08
> 8.00
> >    3.78  133.13  216.40  100.47  14.25  40.48
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               0.59   467.12  191.98    5.28    11.99     1.84
> 143.64
> >   64.63  355.47  359.03  226.22   5.02  99.10
> > dm-0              0.00     0.00  174.76    2.54    10.82     0.01
> 125.05
> >   64.04  392.46  396.73   99.38   5.59  99.10
> > dm-1              0.00     0.00    0.00  469.67     0.00     1.83
> 8.00
> >  129.15  274.97    0.00  274.97   0.26  12.05
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               4.91     0.41  228.02    0.82    12.08     0.00
> 108.15
> >   59.24  241.96  242.07  210.00   4.37  99.96
> > dm-0              0.00     0.00  218.00    1.02    12.97     0.00
> 121.31
> >   52.74  223.31  223.21  244.00   4.56  99.96
> > dm-1              0.00     0.00   28.43    0.00     0.11     0.00
> 8.00
> >    9.19  299.22  299.22    0.00  25.76  73.21
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               5.20     0.60  233.40    1.20    13.55     0.01
> 118.34
> >   81.72  316.28  317.00  177.33   4.26 100.00
> > dm-0              0.00     0.00  243.80    1.00    14.17     0.01
> 118.60
> >   81.06  302.02  302.52  180.80   4.08  99.92
> > dm-1              0.00     0.00    9.60    0.80     0.04     0.00
> 8.00
> >    8.72  496.15  522.75  177.00  96.15 100.00
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               5.60     0.80  205.20    1.80    13.77     0.01
> 136.35
> >  120.57  559.97  562.99  215.11   4.83 100.00
> > dm-0              0.00     0.00  203.60    1.80    13.86     0.01
> 138.27
> >  120.66  565.62  568.50  239.56   4.87 100.00
> > dm-1              0.00     0.00    4.80    0.60     0.02     0.00
> 8.00
> >   12.45 2189.93 2434.83  230.67 136.44  73.68
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda              12.20     0.20  208.00    1.60    13.31     0.01
> 130.18
> >  102.09  540.08  542.48  228.00   4.77 100.00
> > dm-0              0.00     0.00  196.40    1.40    13.09     0.01
> 135.61
> >  100.83  563.25  565.48  249.71   5.02  99.36
> > dm-1              0.00     0.00   25.00    0.20     0.10     0.00
> 8.00
> >   15.92  790.03  794.46  236.00  39.68 100.00
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               8.40     0.60  210.20    2.60    12.13     0.02
> 116.87
> >  101.56  443.61  445.35  302.77   4.70 100.00
> > dm-0              0.00     0.00  189.40    2.20    12.12     0.01
> 129.66
> >  104.66  509.39  511.55  324.00   5.22 100.00
> > dm-1              0.00     0.00   27.80    0.80     0.11     0.00
> 8.00
> >   11.54  386.55  391.17  226.00  34.97 100.00
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               7.73    73.54  206.87    2.06    13.89     0.26
> 138.64
> >  113.54  587.97  591.64  219.33   4.79 100.07
> > dm-0              0.00     0.00  190.21    1.03    12.46     0.00
> 133.43
> >  111.91  633.26  634.96  319.33   5.23 100.07
> > dm-1              0.00     0.00    7.22   74.57     0.03     0.29
> 8.00
> >   24.64  295.01 2199.43  110.71  12.24 100.07
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda              22.01   207.89  233.49   14.35    10.57     0.92
>  94.90
> >   63.80  245.64  258.79   31.80   4.03  99.90
> > dm-0              0.00     0.00  162.44    0.00    10.66     0.00
> 134.42
> >   54.57  328.21  328.21    0.00   6.09  98.95
> > dm-1              0.00     0.00  100.00  222.01     0.39     0.87
> 8.00
> >   24.66   77.74  113.21   61.76   3.10  99.90
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sda               8.00     0.80  185.40    1.00    12.21     0.01
> 134.20
> >   36.76  195.85  196.16  136.80   5.36 100.00
> > dm-0              0.00     0.00  180.40    1.20    12.10     0.00
> 136.56
> >   32.82  182.95  183.42  112.00   5.51 100.00
> > dm-1              0.00     0.00    9.60    0.40     0.04     0.00
> 8.00
> >   17.82  420.32  428.67  220.00  99.60  99.60
> >
> > On 13 February 2012 23:43, Bharath Ravi <bharathravi1@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> I have a distributed HBase setup, on which I'm running the YCSB<
> https://github.com/brianfrankcooper/YCSB/wiki/running-a-workload
> >benchmark.
> >> There are 5 region servers, each a Dual core with around 4GB of memory,
> >> connected simply by a 1Gbps ethernet switch.
> >>
> >> The number of "handlers" per regionserver is set to 500 (!) and HDFS's
> >> maximum receivers per datanode is 4096.
> >>
> >>  The benchmark dataset is large enough not to fit in memory.
> >> Update/Insert/Write throughput goes up to 8000 ops/sec easily.
> >> However, I see read latencies in the order of seconds, and read
> >> throughputs of only a few 100 ops per second.
> >>
> >> "Top" tells me that the CPU's on regionservers spend 70-80% of their
> time
> >> waiting for IO, while disk and network
> >> have plenty of unused bandwidth. How could I diagnose where the read
> >> bottleneck is?
> >>
> >> Any help would be greatly appreciated :)
> >>
> >> Thanks in advance!
> >> --
> >> Bharath Ravi
> >>
> >>
> >
> >
> > --
> > Bharath Ravi
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Bharath Ravi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message