accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Write or Ingest bottleneck
Date Thu, 01 Dec 2016 16:27:22 GMT
No worries on English. It's just difficult to say "I'm not sure what you 
meant" :)

I think your expectations are wrong for your hardware. One 7200rpm SATA 
drive is not going to reach 300K entries/sec ingest rate with Accumulo 
and HDFS. For these specs, 90K entries/sec on one tserver sounds pretty 
good to me. I'd suggest that you add a few more nodes if you want to 
further increase the cluster-wide performance and take advantage of 
near-linear scalability of the system.

hujs wrote:
>        I am so sorry that I did not provide you with more detailed information
> and Bad English, which made it difficult for you to understand.My tServer
> can only reach 900,000entry / s in very few cases. I expect my cluster
> average ingest rate to reach  300,000entry / s. I wanted to say, "Do I need
> to expand my cluster to meet my high ingest rate needs?" This time I list my
> hardware as detailed as possible, and when I'm doing the data insertion, the
> computer Of the cpu, io load is also provided.My cluster contains four
> computers, the computer's CPU is 4 cores, and 8 hardware threads. The
> network is an internal network with a bandwidth of 1 gb / s. Each computer
> contains a hard disk, hard drive model is SATA, Capacity: 1TB, 7200 RPM,
> Firmware: CC43.The following is the acquisition of cpu and disk IO  the
> information when I do insert. This information should be sufficient to
> indicate the status of my cluster most of the time.
>    CPU information in 4 computers:
>    tSever1
> top - 17:42:20 up 9 days,  4:16, 10 users,  load average: 0.88, 0.40, 0.20
> Tasks: 239 total,   1 running, 237 sleeping,   1 stopped,   0 zombie
> %Cpu0  :  2.8 us,  1.8 sy,  0.0 ni, 94.3 id,  0.0 wa,  0.0 hi,  1.1 si,  0.0
> st
> %Cpu1  :  5.7 us,  2.7 sy,  0.0 ni, 91.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu2  :  4.6 us,  1.7 sy,  0.0 ni, 93.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
> st
> %Cpu3  :  2.0 us,  2.7 sy,  0.0 ni, 90.9 id,  4.4 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu4  :  2.3 us,  0.7 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu5  :  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu6  :  0.3 us,  1.0 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu7  :  0.7 us,  1.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> KiB Mem:  24432228 total, 24246836 used,   185392 free,        0 buffers
> KiB Swap: 16457724 total,  2495092 used, 13962632 free.  1016352 cached Mem
> tServer2
> [hadoop@slave13 bin]$ top
> top - 17:43:55 up 15 days,  7:42,  7 users,  load average: 0.84, 0.44, 0.20
> Tasks: 218 total,   2 running, 215 sleeping,   1 stopped,   0 zombie
> %Cpu0  :  2.7 us,  2.1 sy,  0.0 ni, 94.1 id,  0.0 wa,  0.0 hi,  1.1 si,  0.0
> st
> %Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu2  :  3.6 us,  1.6 sy,  0.0 ni, 94.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu3  :  6.3 us,  1.1 sy,  0.0 ni, 92.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu4  :  1.6 us,  0.0 sy,  0.0 ni, 98.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu5  :  0.5 us,  0.0 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu6  :  0.5 us,  0.5 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu7  :  0.5 us,  0.5 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> KiB Mem:  32855648 total, 32619892 used,   235756 free,        0 buffers
> KiB Swap:  8191996 total,    90824 used,  8101172 free. 10842092 cached Mem
>
> tServer3
> [hadoop@slave10 bin]$ top
> top - 17:47:46 up 8 days,  8:10,  9 users,  load average: 0.50, 0.47, 0.35
> Tasks: 286 total,   2 running, 283 sleeping,   1 stopped,   0 zombie
> %Cpu0  : 30.4 us,  1.4 sy,  0.0 ni, 64.3 id,  1.4 wa,  0.0 hi,  2.4 si,  0.0
> st
> %Cpu1  :  5.5 us,  2.8 sy,  0.0 ni, 91.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu2  :  5.0 us,  3.7 sy,  0.0 ni, 89.4 id,  1.4 wa,  0.0 hi,  0.5 si,  0.0
> st
> %Cpu3  :  5.5 us,  1.8 sy,  0.0 ni, 92.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu4  :  6.9 us,  1.4 sy,  0.0 ni, 91.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu5  :  2.3 us,  1.4 sy,  0.0 ni, 92.2 id,  4.1 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu6  :  1.8 us,  1.4 sy,  0.0 ni, 92.2 id,  4.6 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu7  : 11.4 us,  0.5 sy,  0.0 ni, 88.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> KiB Mem:  32689760 total, 32455764 used,   233996 free,        0 buffers
> KiB Swap: 16457724 total,  4678984 used, 11778740 free.   712908 cached Mem
>
> Master
> top - 17:41:37 up 15 days,  7:41, 14 users,  load average: 0.47, 0.68, 0.66
> Tasks: 260 total,   1 running, 259 sleeping,   0 stopped,   0 zombie
> %Cpu0  :  2.1 us,  1.0 sy,  0.0 ni, 96.5 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
> st
> %Cpu1  : 12.0 us,  1.0 sy,  0.0 ni, 87.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu2  :  5.4 us,  1.0 sy,  0.0 ni, 93.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu3  : 12.0 us,  0.3 sy,  0.0 ni, 87.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu4  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu5  :  0.7 us,  0.0 sy,  0.0 ni, 99.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu6  :  1.0 us,  0.3 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> %Cpu7  :  0.7 us,  0.0 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0
> st
> KiB Mem:  32855644 total, 30707652 used,  2147992 free,        0 buffers
> KiB Swap:  8191996 total,   163916 used,  8028080 free. 11263484 cached Mem
>    IO information in 4 computers:
> tServer1
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00  289.00  105.00  3208.00 43904.00   239.15
> 11.50   90.60    5.01  326.20   1.22  48.00
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00  289.00    0.00  3208.00     0.00    22.20
> 12.30  133.56    5.01    0.00   1.66  48.00
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     4.00    2.00  195.00   256.00 88960.00   905.75
> 42.31  214.77   11.50  216.85   2.73  53.80
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    2.00  199.00   256.00 88960.00   887.72
> 43.07  214.30   11.50  216.34   2.68  53.80
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    12.00    8.00  187.00  2304.00 57052.00   608.78
> 44.26  106.96   38.62  109.88   3.52  68.70
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00   13.00     0.00    92.00    14.15
> 2.67    0.00    0.00    0.00  18.08  23.50
> dm-2              0.00     0.00    8.00  263.00  2304.00 87660.00   663.94
> 43.51   78.53   38.75   79.74   2.54  68.80
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00  294.00   83.00  1332.00 30700.00   169.93
> 11.59   92.82    2.22  413.73   1.48  55.80
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    1.00    0.00     4.00     0.00     8.00
> 3.47 6141.00   22.00    0.00 424.00  42.40
> dm-2              0.00     0.00  293.00    0.00  1328.00     0.00     9.06
> 8.87  106.13    2.16    0.00   1.83  53.60
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    13.00    7.00  200.00  2304.00 87428.00   866.98
> 78.27  376.22   46.86  387.75   4.62  95.60
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    7.00  212.00  2304.00 87556.00   820.64
> 83.63  380.09   46.86  391.09   4.37  95.60
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    10.00    2.00  145.00   256.00 63412.00   866.23
> 44.13  140.65   23.00  142.28   4.66  68.50
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    2.00  220.00   256.00 95828.00   865.62
> 46.35   98.66   23.00   99.35   3.09  68.50
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    7.00   74.00  2180.00 32548.00   857.48
> 8.62  400.89  136.57  425.89   7.57  61.30
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    1.00     0.00     4.00     8.00
> 0.39  391.00    0.00  391.00 391.00  39.10
> dm-2              0.00     0.00    7.00    0.00  2180.00     0.00   622.86
> 8.79 4804.57  136.57    0.00  78.86  55.20
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     7.00    2.00  136.00   256.00 59516.00   866.26
> 72.16  246.76   12.00  250.21   4.51  62.20
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    2.00  205.00   256.00 88996.00   862.34
> 75.82  171.81   12.00  173.37   3.00  62.20
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     2.00    8.00  159.00  2176.00 69112.00   853.75
> 39.73  329.22   42.62  343.64   3.59  60.00
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    8.00  201.00  2176.00 91212.00   893.67
> 40.99  278.49   42.62  287.88   2.87  60.00
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.00     0.00  341.00  121.00  1804.00 51728.00   231.74
> 27.95  109.98    4.51  407.20   2.08  95.90
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00  342.00    9.00  1804.00   148.00    11.12
> 28.27  146.20    5.42 5495.67   2.73  95.90
> tServer2:
>
> Device:
>           rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz
> await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda
>                0.00     0.00   24.00    0.00  1956.00     0.00   163.00
> 0.08    3.46    3.46    0.00   0.21   0.50
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    2.00     0.00    20.00    20.00
> 0.02   11.00    0.00   11.00  11.00   2.20
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00   24.00    0.00  1956.00     0.00   163.00
> 0.08    3.29    3.29    0.00   0.21   0.50
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     3.00    0.00  165.00     0.00 74048.00   897.55
> 39.60  240.03    0.00  240.03   2.93  48.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00   25.00    0.00  1952.00     0.00   156.16
> 0.10    4.08    4.08    0.00   0.24   0.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     1.00    0.00  180.00     0.00 91144.00  1012.71
> 1423.88  340.62    0.00  340.62   4.08  73.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00   24.00  214.00  1956.00 107028.00   915.83
> 1842.99 1086.59   93.71 1197.94   4.20 100.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00  210.00     0.00 104996.00   999.96
> 1637.43 2223.34    0.00 2223.34   4.76 100.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00   24.00  258.00  1956.00 129556.00   932.71
> 1405.20 2954.09   75.62 3221.85   3.55 100.00
> tServer3
> Linux 3.10.0-123.el7.x86_64 (slave10) 	2016年12月01日 	_x86_64_	(8 CPU)
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.16     3.55   15.41   15.49   380.75  3379.64   243.43
> 0.17    5.31   28.70  366.79   3.70  11.42
> dm-0              0.00     0.00    1.33    2.70     5.31    10.81     8.00
> 1.92  475.91   10.51  704.81   0.58   0.23
> dm-1              0.00     0.00    0.46    0.60    19.22     4.15    44.33
> 0.14  133.93   44.25  202.19  16.87   1.78
> dm-2              0.00     0.00   14.83   14.22   356.14  3364.75   256.21
> 0.33   11.46   29.21  412.10   3.86  11.21
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     5.00    1.00  196.00     8.00 69136.00   701.97
> 67.54  443.77    8.00  445.99   4.76  93.70
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    1.00    0.00     8.00     0.00    16.00
> 0.01    8.00    8.00    0.00   8.00   0.80
> dm-2              0.00     0.00    0.00  215.00     0.00 88668.00   824.82
> 70.46  437.68    0.00  437.68   4.36  93.70
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    12.00    0.00  220.00     0.00 102300.00   930.00
> 54.77  240.45    0.00  240.45   2.66  58.50
> dm-0              0.00     0.00    0.00    9.00     0.00    36.00     8.00
> 0.01    0.89    0.00    0.89   0.11   0.10
> dm-1              0.00     0.00    0.00    2.00     0.00     8.00     8.00
> 0.59  294.50    0.00  294.50 147.50  29.50
> dm-2              0.00     0.00    0.00  159.00     0.00 67708.00   851.67
> 56.19  337.38    0.00  337.38   3.67  58.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00  100.00     0.00 34068.00   681.36
> 11.16  276.67    0.00  276.67   6.42  64.20
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    0.00   11.00     0.00  1728.00   314.18
> 11.45 2631.27    0.00 2631.27  58.45  64.30
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     7.00    0.00  179.00     0.00 72580.00   810.95
> 22.02  123.01    0.00  123.01   4.15  74.20
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    0.00  182.00     0.00 72580.00   797.58
> 22.76  125.04    0.00  125.04   4.08  74.20
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    13.00   72.00  143.00   292.00 60944.00   569.64
> 19.76   91.92    4.47  135.95   2.11  45.40
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00   72.00  156.00   292.00 60944.00   537.16
> 22.46   98.50    4.47  141.90   1.99  45.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     8.00   47.00  135.00   288.00 60496.00   667.96
> 35.52   50.65    8.62   65.29   3.15  57.40
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    5.00    0.00    96.00     0.00    38.40
> 0.14   27.00   27.00    0.00   5.60   2.80
> dm-2              0.00     0.00   42.00  239.00   192.00 93776.00   668.81
> 37.35   32.75    6.43   37.38   1.94  54.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    15.00    0.00  152.00     0.00 33692.00   443.32
> 42.97  427.57    0.00  427.57   4.86  73.80
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    0.00   92.00     0.00   636.00    13.83
> 47.52  765.02    0.00  765.02   8.02  73.80
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     3.00    5.00  213.00    32.00 80012.00   734.35
> 44.18  222.28  134.80  224.33   4.44  96.80
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    5.00    0.00    32.00     0.00    12.80
> 0.67  134.80  134.80    0.00  85.20  42.60
> dm-2              0.00     0.00    0.00  180.00     0.00 79788.00   886.53
> 46.19  286.00    0.00  286.00   5.38  96.80
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    13.00   45.00  107.00   748.00 41388.00   554.42
> 21.19   31.96    2.36   44.41   2.34  35.50
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    1.00     0.00     4.00     8.00
> 0.02   20.00    0.00   20.00  20.00   2.00
> dm-2              0.00     0.00   46.00  229.00   752.00 84284.00   618.44
> 22.67   18.83    2.28   22.15   1.21  33.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     4.00    6.00  134.00   124.00 54676.00   782.86
> 23.53  254.96   20.00  265.48   2.12  29.70
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    5.00  157.00   120.00 68252.00   844.10
> 25.72  240.25   24.00  247.14   1.83  29.70
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00  149.00     0.00 56756.00   761.83
> 31.05  236.34    0.00  236.34   4.77  71.00
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    0.00   11.00     0.00   280.00    50.91
> 32.82 3373.82    0.00 3373.82  64.55  71.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    11.00    1.00  154.00    80.00 64852.00   837.83
> 52.95  305.46   27.00  307.27   4.75  73.70
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    1.00    0.00    80.00     0.00   160.00
> 0.03   27.00   27.00    0.00  27.00   2.70
> dm-2              0.00     0.00    0.00  169.00     0.00 68444.00   809.99
> 56.57  301.54    0.00  301.54   4.20  71.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     8.00    0.00  139.00     0.00 65104.00   936.75
> 22.07  154.24    0.00  154.24   2.32  32.30
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    0.00  160.00     0.00 67772.00   847.15
> 23.14  136.97    0.00  136.97   2.02  32.30
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    14.00   58.00  119.00   252.00 53956.00   612.52
> 28.45  105.42    7.66  153.07   2.69  47.60
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00   59.00  168.00   272.00 67900.00   600.63
> 32.59   85.42    7.53  112.77   2.10  47.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     4.00    2.00   79.00    24.00 31980.00   790.22
> 8.54  242.48   54.00  247.25   3.80  30.80
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    1.00    0.00     4.00     0.00     8.00
> 0.02   24.00   24.00    0.00  24.00   2.40
> dm-2              0.00     0.00    0.00  152.00     0.00 68188.00   897.21
> 8.81  156.31    0.00  155.75   1.88  28.50
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.00     0.00  116.00  137.00   772.00 56828.00   455.34
> 24.63  116.15    9.72  206.26   2.99  75.60
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    6.00    0.00   176.00     0.00    58.67
> 0.24   39.33   39.33    0.00  20.17  12.10
> dm-2              0.00     0.00  113.00    8.00   604.00   420.00    16.93
> 25.71  253.08    8.05 3714.12   6.25  75.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00    12.00   16.00  180.00    64.00 67584.00   690.29
> 42.08  215.60    3.56  234.45   4.43  86.90
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00   14.00  184.00    56.00 67580.00   683.19
> 46.02  233.32    4.07  250.77   4.39  86.90
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     4.00    0.00  158.00     0.00 68420.00   866.08
> 26.00  164.56    0.00  164.56   2.46  38.90
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-2              0.00     0.00    0.00  162.00     0.00 68420.00   844.69
> 27.00  166.70    0.00  166.70   2.40  38.90
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     9.00    2.00  139.00   124.00 66968.00   951.66
> 19.16  135.77   35.50  137.21   2.28  32.10
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> dm-1              0.00     0.00    2.00    0.00   124.00     0.00   124.00
> 0.07   35.50   35.50    0.00  18.00   3.60
> dm-2              0.00     0.00    1.00  148.00     4.00 66968.00   898.95
> 20.72  138.99    0.00  139.93   1.92  28.60
>
> Master
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.08     1.17    9.62   28.36   510.84  7320.01   412.36
> 1.54   40.53   19.37   47.71   3.29  12.50
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    2.00  101.00    16.00 47456.00   921.79
> 9.17   89.04    3.50   90.73   3.50  36.10
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    5.00     0.00    48.00    19.20
> 0.05   10.00    0.00   10.00  10.00   5.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    5.00     0.00    48.00    19.20
> 0.04    9.00    0.00    9.00   9.00   4.50
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     2.00    0.00   21.00     0.00  3912.00   372.57
> 0.17    8.24    0.00    8.24   5.90  12.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    4.00     0.00     8.00     4.00
> 0.03    6.50    0.00    6.50   6.50   2.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    8.00     0.00   104.00    26.00
> 0.09   11.50    0.00   11.50   7.62   6.10
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     1.00    0.00  104.00     0.00 50500.00   971.15
> 43.53   89.70    0.00   89.70   2.98  31.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     1.00    0.00  174.00     0.00 85056.00   977.66
> 37.63  412.84    0.00  412.84   3.42  59.50
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    1.00    3.00     4.00    28.00    16.00
> 0.03    8.50   18.00    5.33   8.50   3.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    3.00     0.00    60.00    40.00
> 0.04   11.67    0.00   11.67  11.67   3.50
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00   11.00     0.00  3960.00   720.00
> 0.13   12.18    0.00   12.18   7.82   8.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    3.00     0.00     8.00     5.33
> 0.02    4.67    0.00    4.67   6.00   1.80
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00   13.00     0.00   288.00    44.31
> 0.18   13.92    0.00   13.92  10.92  14.20
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    6.00     0.00  1788.00   596.00
> 0.02    3.00    0.00    3.00   0.67   0.40
>     So,What should I do?
>                        thank you very much!
>
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Write-or-Ingest-bottleneck-tp19255p19330.html
> Sent from the Developers mailing list archive at Nabble.com.

Mime
View raw message