Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jgray@facebook.com designates
 69.63.179.25 as permitted sender)
From: Jonathan Gray <jgray@facebook.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Date: Tue, 6 Apr 2010 15:15:06 -0700
Subject: RE: how can I check the I/O influence HBase to HDFS
Thread-Topic: how can I check the I/O influence HBase to HDFS
Thread-Index: AcrVnrUkou2xe+oPRRGhP45QS3cGCQAN3vwA
Message-ID: 
 <8D66B74984F9564BBB25C3C67D630F2D6689CF3E@SC-MBXC1.TheFacebook.com>
References: <l2x6ba3573d1004060834n7b543e0rd0d5b7cdacb84f9b@mail.gmail.com>
In-Reply-To: <l2x6ba3573d1004060834n7b543e0rd0d5b7cdacb84f9b@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Can you explain more about what information you are trying to find out?

You had an existing HDFS and you want to measure the additional impact addi=
ng HBase is?  Is that in terms of reads/writes/iops or data size?

If you have a steady-state set of metrics for HDFS w/o HBase, can you not j=
ust monitor those metrics w/ HBase running and calculate the deltas?

Also, to what end are you trying to figure this out?  I'm very much interes=
ted in what courses of actions you might take given the different informati=
on you could find out about HBase's influence on your cluster.

JG

> -----Original Message-----
> From: steven zhuang [mailto:steven.zhuang.1984@gmail.com]
> Sent: Tuesday, April 06, 2010 8:34 AM
> To: hbase-user@hadoop.apache.org
> Subject: how can I check the I/O influence HBase to HDFS
>=20
> hi, there,
>               I have this problem of checking the influence HBase
> brought to
> HDFS.
>               I have a Hadoop cluster which has 30+ data nodes, and a
> Hbase
> cluster based on it, with 18 regionservers residing on 18 datanodes.
>               we have observed the HDFS IO has increased a lot if we do
> some
> importing or query ops on hbase tables, but we don't know how
> much would hbase impact the HDFS, so now I have to dig into this.
>               my idea is as follows:
>=20
>                  1.  grep from regionservers logs the file information
> of
> hbase tables, which mainly should be store files' names and their
> sizes, sum
> the size up.
>                  2. grep from datanodes' logs the HDFS_READ/HDFS_WRITE
> log,
> and calculate the whole IO bytes.
>                  3. get the rate of HBase IO / HDFS IO.
>=20
>                my concern is that if the above idea is right, is there
> anything missing or a better way to do this?
>=20
>                And to make it more convinsible, I want to have the
> block
> info for each HTable's, not just those ones under each table's
> directory,
> but also those store files which was later removed by major compaction,
> since in datanode log, all I can see is block id, any pointer or hint
> is
> really appreciated.