Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of dean.hiller@broadridge.com
 designates 64.18.2.215 as permitted sender)
From: "Hiller, Dean  x66079" <dean.hiller@broadridge.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Date: Sun, 5 Jun 2011 19:21:40 -0400
Subject: performance monitoring question
Thread-Topic: performance monitoring question
Thread-Index: Acwj11LDJVfsA7BZQl+OlM0IrkyFIQ==
Message-ID: 
 <08230D4C8E666D479F6DE495A7684DCD0AABC05E@JSCPCWEXMAA1.bsg.ad.adp.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_08230D4C8E666D479F6DE495A7684DCD0AABC05EJSCPCWEXMAA1bsg_"
MIME-Version: 1.0

--_000_08230D4C8E666D479F6DE495A7684DCD0AABC05EJSCPCWEXMAA1bsg_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

So, we were loading from a single client and had a queue.  The queue kept g=
etting up to the fill level even though 20 threads are emptying the queue w=
hile 1 thread is pushing into the queue implying writing to hbase was makin=
g it fill up my queue.  Checking cpu and iostat for disk utilization on the=
 client, we don't really see the client as a bottleneck, but this fights ev=
ery bone in my body so I ran the client on two nodes to process half the it=
ems it was processing to begin with and sure enough, it doubled the speed a=
nd that node was a bottleneck.

I keep thinking maybe the Bus was at 100% utilization or something that is =
not measurable.  The disk was not really showing anything vs. when we make =
hadoop map/reduce against Sybase and we see the disk utilization go to 100%=
 making it easy to spot.  Anyone have any experience with monitoring at all=
 and good stats to capture.

We were looking at "iostat -x 5 25" on linux to see how cpu and disk were d=
oing, and then measuring the network which is near-zero right now as well a=
s monitoring memory.  Anyone else have some good commands to make it easier=
 to spot a bottleneck?

Thanks,
Dean


This message and any attachments are intended only for the use of the add=
ressee and
may contain information that is privileged and confidential. If the reade=
r of the =

message is not the intended recipient or an authorized representative of =
the
intended recipient, you are hereby notified that any dissemination of thi=
s
communication is strictly prohibited. If you have received this communica=
tion in
error, please notify us immediately by e-mail and delete the message and =
any
attachments from your system.
=0D
--_000_08230D4C8E666D479F6DE495A7684DCD0AABC05EJSCPCWEXMAA1bsg_--