hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kannan Muthukkaruppan <Kan...@facebook.com>
Subject RE: RPC statistics from region servers...
Date Thu, 29 Jul 2010 22:16:12 GMT
Pushed the hadoop-metrics.properties changes, and it seems better but not quite. [It doesn't
look like all the Region Servers are reporting similar set of counters (on some I am seeing
"multiput" stuff, although we aren' t), lot of the average times are close to 0.]

So next I looked into the HBASE-2146 patch, and that doesn't appear to be in trunk although
at some point it was in trunk earlier.

http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402

Will try applying the patch locally to see if that helps.

Regards,
Kannan
-----Original Message-----
From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
Sent: Thursday, July 29, 2010 2:41 PM
To: dev@hbase.apache.org
Subject: RE: RPC statistics from region servers...

Gary,

With some recent restructuring of our conf/* and cluster deploy scripts, we did lose our hadoop-metrics.properties
changes, and reverted back to the default one.  That could very well report the issue I ran
into. I'll report back when I try it out.

Thanks again for your help on this.

Q: Why don't we make the suggested settings in http://hbase.apache.org/docs/current/metrics.html
be part of the default conf/hadoop-metrics.properties?

Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:ghelmling@gmail.com]
Sent: Thursday, July 29, 2010 1:08 AM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...

Hmm, this sounds like HBASE-2146.  That patch had added code to initialize
the RPC metrics MBean with a know list of RPC method names maintained in
HBaseRPC.Invocation.  That way all per-method metrics would show up on
server startup.  I believe that that method name mapping was later removed
though (it can make the RPC serialization brittle, complicating things like
rolling restarts), so maybe the patch that was applied to trunk was later
removed?

The MBean attributes for individual RPC methods _should_ be there after
there has been activity involving those requests against the cluster.  If
you start jconsole prior to the cluster activity however, I don't think the
MBean definition will update, so you'll need to restart jconsole for them to
show up.

In addition, you should make sure you're running with a metrics context with
an updater configured in
conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
GangliaContext, etc).  And configure rpc.period with the update interval.
These metrics don't show up as live counters in JMX, they're only pushed out
to JMX during the periodic updates.

If you've already configured all of those conditions and the metrics _still_
aren't showing up, then let me know and I'll try to do some testing on trunk
tomorrow.

--gh


On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
<Kannan@facebook.com>wrote:

> > they don't exist until used?
>
> I was doing a whole bunch of puts & gets against the cluster during that
> time. That's what you mean, correct?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Wednesday, July 28, 2010 8:21 PM
> To: dev@hbase.apache.org
> Cc: hbase-dev@hadoop.apache.org
> Subject: Re: RPC statistics from region servers...
>
> I think those are dynamic metrics... they don't exist until used?
>
> On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Kannan@facebook.com>
> wrote:
> > I was using jconsole to check the stats published by region servers..
> >
> > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> >
> > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > and,
> > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> >
> > I no longer see attributes like:
> > "incrementColumnValueNumOps "
> > "incrementColumnValueAvgTime "
> > "putNumOps "
> > "putAvgTime "
> > "getNumOps "
> > "getAvgTime "
> > "deleteNumOps "
> > "deleteAvgTime "
> > "openScannerNumOps "
> > "openScannerAvtTime "
> > "nextNumOps "
> > "nextAvgTime "
> > "closeNumOps "
> > "closeAvgTime "
> >
> > Anyone aware of what change happened in this area?
> >
> > regards,
> > Kannan
> >
> >
>

Mime
View raw message