Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0155C11817 for ; Thu, 12 Jun 2014 01:38:49 +0000 (UTC) Received: (qmail 99029 invoked by uid 500); 12 Jun 2014 01:38:47 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 98953 invoked by uid 500); 12 Jun 2014 01:38:46 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 98942 invoked by uid 99); 12 Jun 2014 01:38:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jun 2014 01:38:46 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of fancyerii@gmail.com designates 209.85.215.48 as permitted sender) Received: from [209.85.215.48] (HELO mail-la0-f48.google.com) (209.85.215.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jun 2014 01:38:41 +0000 Received: by mail-la0-f48.google.com with SMTP id gl10so309035lab.35 for ; Wed, 11 Jun 2014 18:38:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=lRh/KJXelwTIBEKCz7AigMeaZNwAt5mJpZMyCe02NLo=; b=0u87TLx7IUXMkcUmBBwRVi1I0lopQzjt/39wbbLR3sBG40PFcp93D14kEAG+jMLLPX 4O2tyRHpRnxMI0Quxqo0ByddmxWExLgefY2L9xvSP4RDtS9n38hzyEg22qnAkMpquci2 BnRj6lOaqkP09odLg+AM3tINK8ofktF3vfFOlCe/MhSgeU4h2YR8qN0Hte6db0Ptyaeh NplGDNg3mRbsF2+ZouXC7goiP/Ve6sTpK3IVdba97SuG6Q4fdV7poKwaoCRJ0Rcjl2w0 rshRVxmkNAXcjoBD6lc2lIjGEZTFg61j0aEKJYFJbQqG7MBDdbBwSMnETAD8UzifX/WE Gz6A== MIME-Version: 1.0 X-Received: by 10.112.73.234 with SMTP id o10mr13841728lbv.41.1402537099923; Wed, 11 Jun 2014 18:38:19 -0700 (PDT) Received: by 10.112.151.200 with HTTP; Wed, 11 Jun 2014 18:38:19 -0700 (PDT) Date: Thu, 12 Jun 2014 09:38:19 +0800 Message-ID: Subject: high load average in one region server From: Li Li To: user@hbase.apache.org Content-Type: text/plain; charset=UTF-8 X-Virus-Checked: Checked by ClamAV on apache.org I have 5 region server hbase cluster. today I found one rs server's load average is above 100 while the other 4 is less than 1. I use vmstat and dstat and found that this high load machine have large number of read(about 30M/s) and network sent. Does that mean the cluster suffers hot spot? the slow machine is mphbase2 1. base statistics ServerName Start time Requests Per Second Num. Regions mphbase1,60020,1402298228045 Mon Jun 09 15:17:08 CST 2014 586 35 mphbase2,60020,1402298228527 Mon Jun 09 15:17:08 CST 2014 539 32 mphbase3,60020,1402298228361 Mon Jun 09 15:17:08 CST 2014 966 32 mphbase4,60020,1402298159826 Mon Jun 09 15:15:59 CST 2014 518 35 mphbase5,60020,1402298228382 Mon Jun 09 15:17:08 CST 2014 442 36 Total:5 3051 170 2. storefiles ServerName Num. Stores Num. Storefiles Storefile Size Uncompressed Storefile Size Index Size Bloom Size mphbase1,60020,1402298228045 35 67 11872m 11874mb 8783k 34818k mphbase2,60020,1402298228527 32 61 11976m 11977mb 8882k 34976k mphbase3,60020,1402298228361 32 66 18321m 18325mb 13470k 54872k mphbase4,60020,1402298159826 35 72 13842m 13848mb 10753k 31784k mphbase5,60020,1402298228382 36 78 15021m 15027mb 15329k 29321k 3. hdfs info(from hdfs) Live Datanodes : 5 Node Last Contact Admin Stat Configured Capacity (GB) Used (GB) Non DFS Used (GB) Remaining (GB) Used(%) Used(%) Remaining(%) Blocks mphbase1 0 In Service 457.55 53.28 52.23 352.04 11.64 76.94 1150 mphbase2 1 In Service 457.55 46.56 48.89 362.1 10.18 79.14 971 mphbase3 0 In Service 457.55 52.05 55.6 349.89 11.38 76.47 1128 mphbase4 1 In Service 457.55 50.25 36.88 370.42 10.98 80.96 1254 mphbase5 2 In Service 457.55 55.2 49.29 353.06 12.06 77.16 1338