hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Improving HBase read performance (based on YCSB)
Date Wed, 15 Feb 2012 03:39:22 GMT
Yep, definitely bound on seeks - see the 100% util, and the r/s >100.
The bandwidth provided by random IO from a disk is going to be much
smaller than the sequential IO you see from hdparm

-Todd

On Tue, Feb 14, 2012 at 3:06 PM, Bharath Ravi <bharathravi1@gmail.com> wrote:
> Thanks Todd!
>
> I check disk bandwidth by first running "hparm" on it, (this shows me a
> read b/w of around 56Mbps)
> and then running "iftop" while the benchmarks run (This shows me that reads
> are only around 10-15Mbps: but
> this could definitely be because random seeks are a bottleneck)
>
> The iostat output seems to suggest seek is a problem too, although I'm not
> sure I interpret these numbers correctly.
> Here's some output from iostat, while the benchmark runs:
>
> Do the queue-lengths I see here indicate a bottleneck?
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.00     1.28    6.31    2.57     0.43     1.01  
332.30
>    3.63  408.90  259.25  776.48   6.40   5.67
> dm-0              0.00     0.00    6.55    2.87     0.43     1.01  
311.48
>    4.60  487.90  380.14  733.49   5.99   5.65
> dm-1              0.00     0.00    0.29    0.88     0.00     0.00  
  8.00
>    1.33 1135.17   89.17 1479.15   3.23   0.38
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.40    18.20  257.80    2.60    13.84     0.08  
109.49
>   62.55  240.45  241.44  141.85   4.43 115.28
> dm-0              0.00     0.00  258.80    3.40    13.81     0.01  
107.99
>   63.17  241.17  241.93  183.76   4.40 115.28
> dm-1              0.00     0.00    0.00   17.20     0.00     0.07  
  8.00
>    0.21   12.00    0.00   12.00   0.14   0.24
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.60   255.80  234.60    3.00    16.43     1.01   150.36
>  113.11  427.74  430.83  186.40   4.82 114.56
> dm-0              0.00     0.00  262.00    1.00    18.03     0.00  
140.44
>  113.86  389.06  389.87  175.20   4.36 114.56
> dm-1              0.00     0.00    0.20  258.00     0.00     1.01  
  8.00
>   37.92  146.87    0.00  146.98   1.02  26.32
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               3.42    98.67  152.09    2.66     9.47     0.40  
130.68
>   75.82  543.73  549.56  210.86   4.61  71.33
> dm-0              0.00     0.00  132.32    1.33     8.04     0.01  
123.43
>   76.06  631.83  635.09  308.00   5.34  71.33
> dm-1              0.00     0.00    3.04   99.62     0.01     0.39  
  8.00
>   14.84  144.57  648.75  129.18   2.72  27.91
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.27    54.85  220.89    2.95    13.15     0.23  
122.44
>   66.15  304.77  305.75  231.71   4.43  99.16
> dm-0              0.00     0.00  232.49    3.38    14.07     0.02  
122.30
>   66.66  291.36  292.25  230.00   4.20  99.16
> dm-1              0.00     0.00    0.00   54.22     0.00     0.21  
  8.00
>   18.12  334.27    0.00  334.27   1.57   8.52
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               4.00    19.80  224.60    3.00    12.80     0.09  
115.95
>   62.69  254.17  256.84   54.67   4.39  99.92
> dm-0              0.00     0.00  229.40    2.20    13.00     0.01  
115.01
>   61.95  246.34  247.99   73.82   4.29  99.28
> dm-1              0.00     0.00    8.00   20.40     0.03     0.08  
  8.00
>    3.78  133.13  216.40  100.47  14.25  40.48
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.59   467.12  191.98    5.28    11.99     1.84   143.64
>   64.63  355.47  359.03  226.22   5.02  99.10
> dm-0              0.00     0.00  174.76    2.54    10.82     0.01  
125.05
>   64.04  392.46  396.73   99.38   5.59  99.10
> dm-1              0.00     0.00    0.00  469.67     0.00     1.83  
  8.00
>  129.15  274.97    0.00  274.97   0.26  12.05
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               4.91     0.41  228.02    0.82    12.08     0.00  
108.15
>   59.24  241.96  242.07  210.00   4.37  99.96
> dm-0              0.00     0.00  218.00    1.02    12.97     0.00  
121.31
>   52.74  223.31  223.21  244.00   4.56  99.96
> dm-1              0.00     0.00   28.43    0.00     0.11     0.00  
  8.00
>    9.19  299.22  299.22    0.00  25.76  73.21
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               5.20     0.60  233.40    1.20    13.55     0.01  
118.34
>   81.72  316.28  317.00  177.33   4.26 100.00
> dm-0              0.00     0.00  243.80    1.00    14.17     0.01  
118.60
>   81.06  302.02  302.52  180.80   4.08  99.92
> dm-1              0.00     0.00    9.60    0.80     0.04     0.00  
  8.00
>    8.72  496.15  522.75  177.00  96.15 100.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               5.60     0.80  205.20    1.80    13.77     0.01  
136.35
>  120.57  559.97  562.99  215.11   4.83 100.00
> dm-0              0.00     0.00  203.60    1.80    13.86     0.01  
138.27
>  120.66  565.62  568.50  239.56   4.87 100.00
> dm-1              0.00     0.00    4.80    0.60     0.02     0.00  
  8.00
>   12.45 2189.93 2434.83  230.67 136.44  73.68
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda              12.20     0.20  208.00    1.60    13.31     0.01  
130.18
>  102.09  540.08  542.48  228.00   4.77 100.00
> dm-0              0.00     0.00  196.40    1.40    13.09     0.01  
135.61
>  100.83  563.25  565.48  249.71   5.02  99.36
> dm-1              0.00     0.00   25.00    0.20     0.10     0.00  
  8.00
>   15.92  790.03  794.46  236.00  39.68 100.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               8.40     0.60  210.20    2.60    12.13     0.02  
116.87
>  101.56  443.61  445.35  302.77   4.70 100.00
> dm-0              0.00     0.00  189.40    2.20    12.12     0.01  
129.66
>  104.66  509.39  511.55  324.00   5.22 100.00
> dm-1              0.00     0.00   27.80    0.80     0.11     0.00  
  8.00
>   11.54  386.55  391.17  226.00  34.97 100.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               7.73    73.54  206.87    2.06    13.89     0.26  
138.64
>  113.54  587.97  591.64  219.33   4.79 100.07
> dm-0              0.00     0.00  190.21    1.03    12.46     0.00  
133.43
>  111.91  633.26  634.96  319.33   5.23 100.07
> dm-1              0.00     0.00    7.22   74.57     0.03     0.29  
  8.00
>   24.64  295.01 2199.43  110.71  12.24 100.07
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda              22.01   207.89  233.49   14.35    10.57     0.92    94.90
>   63.80  245.64  258.79   31.80   4.03  99.90
> dm-0              0.00     0.00  162.44    0.00    10.66     0.00  
134.42
>   54.57  328.21  328.21    0.00   6.09  98.95
> dm-1              0.00     0.00  100.00  222.01     0.39     0.87    
8.00
>   24.66   77.74  113.21   61.76   3.10  99.90
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               8.00     0.80  185.40    1.00    12.21     0.01  
134.20
>   36.76  195.85  196.16  136.80   5.36 100.00
> dm-0              0.00     0.00  180.40    1.20    12.10     0.00  
136.56
>   32.82  182.95  183.42  112.00   5.51 100.00
> dm-1              0.00     0.00    9.60    0.40     0.04     0.00  
  8.00
>   17.82  420.32  428.67  220.00  99.60  99.60
>
> On 13 February 2012 23:43, Bharath Ravi <bharathravi1@gmail.com> wrote:
>
>> Hi all,
>>
>> I have a distributed HBase setup, on which I'm running the YCSB<https://github.com/brianfrankcooper/YCSB/wiki/running-a-workload>benchmark.
>> There are 5 region servers, each a Dual core with around 4GB of memory,
>> connected simply by a 1Gbps ethernet switch.
>>
>> The number of "handlers" per regionserver is set to 500 (!) and HDFS's
>> maximum receivers per datanode is 4096.
>>
>>  The benchmark dataset is large enough not to fit in memory.
>> Update/Insert/Write throughput goes up to 8000 ops/sec easily.
>> However, I see read latencies in the order of seconds, and read
>> throughputs of only a few 100 ops per second.
>>
>> "Top" tells me that the CPU's on regionservers spend 70-80% of their time
>> waiting for IO, while disk and network
>> have plenty of unused bandwidth. How could I diagnose where the read
>> bottleneck is?
>>
>> Any help would be greatly appreciated :)
>>
>> Thanks in advance!
>> --
>> Bharath Ravi
>>
>>
>
>
> --
> Bharath Ravi



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message