Return-Path: 
 <user-return-28493-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E1C3D4B3
	for <apmail-hbase-user-archive@www.apache.org>;
 Mon,  9 Jul 2012 18:36:22 +0000 (UTC)
Received: (qmail 6837 invoked by uid 500); 9 Jul 2012 18:36:20 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 6789 invoked by uid 500); 9 Jul 2012 18:36:20 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 6781 invoked by uid 99); 9 Jul 2012 18:36:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Jul 2012 18:36:20 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of asaf.mesika@gmail.com
 designates 209.85.212.179 as permitted sender)
Received: from [209.85.212.179] (HELO mail-wi0-f179.google.com)
 (209.85.212.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Jul 2012 18:36:11 +0000
Received: by wibhq4 with SMTP id hq4so2558724wib.2
        for <user@hbase.apache.org>; Mon, 09 Jul 2012 11:35:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        bh=dvYa5/t2pmPquxIUXoqXI+kR19fEiyGnV7onSI54Xj8=;
        b=BmyUDbBcCmBVFrkBv60q3vzu085QOpk5qp8TWZdc65IzeVSHJ2uXQrmJvixrNqcguG
         WdUQ9a8SYrNqtOHRfbyuhRiCX8MfVgExWR3KjS799niNxhqLw8Ct3cLC5WNjkJZIjYpw
         K7Jm7jLzaQTQxJHP0eLhPz5ojTcsz6SMUcExDFQeIFGH4qw1oB00sgcfOLPd8OcZaA3I
         LBZp+7m3mTNmkNed0nAsLRLgeSupdAAPIXpdIuHbK67nh2e9FI1VC9wqoc4Epwp7drVE
         IfkFfQGDHlZ8WFeguknDLupBK9xdKC2d8smHWBfDjFH9rDWMgQUUMQrdcgEa8kVi4hY6
         7JtQ==
Received: by 10.216.255.73 with SMTP id i51mr8085136wes.46.1341858951185;
        Mon, 09 Jul 2012 11:35:51 -0700 (PDT)
Received: from [192.168.1.103] (109-186-15-10.bb.netvision.net.il.
 [109.186.15.10])
        by mx.google.com with ESMTPS id t8sm24636109wiy.3.2012.07.09.11.35.50
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 09 Jul 2012 11:35:50 -0700 (PDT)
From: Asaf Mesika <asaf.mesika@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: HBase becomes ultra-slaggy
Date: Mon, 9 Jul 2012 21:35:48 +0300
Message-Id: <66B37CC3-E4FD-46D6-84E0-B57B4621D29F@gmail.com>
To: user@hbase.apache.org
Mime-Version: 1.0 (Apple Message framework v1278)
X-Mailer: Apple Mail (2.1278)

Hi,

My cluster started being incredibly slow in the past 2 days.
I've seen many Blocking updates on the region server logs, which lead me =
to believe HDFS creates is the bottleneck.
I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) =
which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, =
when it was previously around 28MB/sec.

I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
And 1 Master computer, hosting NameNode as well.

The slowest computer has Core 2 Duo 2.93GHz, 4GB.

What steps can I take to find out the source of this slowdown?


Thanks!

Asaf Mesika