hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Li <fancye...@gmail.com>
Subject high load average in one region server
Date Thu, 12 Jun 2014 01:38:19 GMT
   I have 5 region server hbase cluster. today I found one rs server's
load average is above 100 while the other 4 is less than 1.
   I use vmstat and dstat and found that this high load machine have
large number of read(about 30M/s) and network sent.
   Does that mean the cluster suffers hot spot? the slow machine is mphbase2
   1. base statistics
  ServerName Start time Requests Per Second Num. Regions
mphbase1,60020,1402298228045 Mon Jun 09 15:17:08 CST 2014 586 35
mphbase2,60020,1402298228527 Mon Jun 09 15:17:08 CST 2014 539 32
mphbase3,60020,1402298228361 Mon Jun 09 15:17:08 CST 2014 966 32
mphbase4,60020,1402298159826 Mon Jun 09 15:15:59 CST 2014 518 35
mphbase5,60020,1402298228382 Mon Jun 09 15:17:08 CST 2014 442 36
Total:5 3051 170

    2. storefiles
ServerName Num. Stores Num. Storefiles Storefile Size Uncompressed
Storefile Size Index Size Bloom Size
mphbase1,60020,1402298228045 35 67 11872m 11874mb 8783k 34818k
mphbase2,60020,1402298228527 32 61 11976m 11977mb 8882k 34976k
mphbase3,60020,1402298228361 32 66 18321m 18325mb 13470k 54872k
mphbase4,60020,1402298159826 35 72 13842m 13848mb 10753k 31784k
mphbase5,60020,1402298228382 36 78 15021m 15027mb 15329k 29321k

    3. hdfs info(from hdfs)
Live Datanodes : 5
Node Last Contact  Admin Stat  Configured Capacity (GB) Used (GB)  Non
DFS Used (GB)  Remaining (GB)  Used(%) Used(%) Remaining(%)  Blocks
mphbase1 0 In Service 457.55 53.28 52.23 352.04 11.64 76.94 1150
mphbase2 1 In Service 457.55 46.56 48.89 362.1 10.18 79.14 971
mphbase3 0 In Service 457.55 52.05 55.6 349.89 11.38 76.47 1128
mphbase4 1 In Service 457.55 50.25 36.88 370.42 10.98 80.96 1254
mphbase5 2 In Service 457.55 55.2 49.29 353.06 12.06 77.16 1338

Mime
View raw message