hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: RPC statistics from region servers...
Date Thu, 29 Jul 2010 22:54:08 GMT
Hi,

I did HBASE-2219, it was a pretty bad hack, since it prevented even
doing a uber-trivial RPC change without ruining background
compatibility, and in my micro-benchmarks, it wasnt even faster to
boot.

I was talking to Kannan earlier on IRC, and he was noting about
multi-put vs put... I noted that nearly all Puts, even of Size=1 go
through the multi-put code path now, and that seemed to be part of the
problem as well.

-ryan

On Thu, Jul 29, 2010 at 3:50 PM, Gary Helmling <ghelmling@gmail.com> wrote:
> Kannan,
>
> It looks like that code went away as part of HBASE-2219, which removed the
> mapping of rpc method names to codes that was previously used in
> serialization.
>
> We could do a similar initialization of the JMX MBean by just introspecting
> the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> HRegionInterface).  Want to open a JIRA?
>
> Gary
>
>
> On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan
> <Kannan@facebook.com>wrote:
>
>> Pushed the hadoop-metrics.properties changes, and it seems better but not
>> quite. [It doesn't look like all the Region Servers are reporting similar
>> set of counters (on some I am seeing "multiput" stuff, although we aren' t),
>> lot of the average times are close to 0.]
>>
>> So next I looked into the HBASE-2146 patch, and that doesn't appear to be
>> in trunk although at some point it was in trunk earlier.
>>
>>
>> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
>>
>> Will try applying the patch locally to see if that helps.
>>
>> Regards,
>> Kannan
>> -----Original Message-----
>> From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
>> Sent: Thursday, July 29, 2010 2:41 PM
>> To: dev@hbase.apache.org
>> Subject: RE: RPC statistics from region servers...
>>
>> Gary,
>>
>> With some recent restructuring of our conf/* and cluster deploy scripts, we
>> did lose our hadoop-metrics.properties changes, and reverted back to the
>> default one.  That could very well report the issue I ran into. I'll report
>> back when I try it out.
>>
>> Thanks again for your help on this.
>>
>> Q: Why don't we make the suggested settings in
>> http://hbase.apache.org/docs/current/metrics.html be part of the default
>> conf/hadoop-metrics.properties?
>>
>> Regards,
>> Kannan
>> -----Original Message-----
>> From: Gary Helmling [mailto:ghelmling@gmail.com]
>> Sent: Thursday, July 29, 2010 1:08 AM
>> To: dev@hbase.apache.org
>> Subject: Re: RPC statistics from region servers...
>>
>> Hmm, this sounds like HBASE-2146.  That patch had added code to initialize
>> the RPC metrics MBean with a know list of RPC method names maintained in
>> HBaseRPC.Invocation.  That way all per-method metrics would show up on
>> server startup.  I believe that that method name mapping was later removed
>> though (it can make the RPC serialization brittle, complicating things like
>> rolling restarts), so maybe the patch that was applied to trunk was later
>> removed?
>>
>> The MBean attributes for individual RPC methods _should_ be there after
>> there has been activity involving those requests against the cluster.  If
>> you start jconsole prior to the cluster activity however, I don't think the
>> MBean definition will update, so you'll need to restart jconsole for them
>> to
>> show up.
>>
>> In addition, you should make sure you're running with a metrics context
>> with
>> an updater configured in
>> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
>> GangliaContext, etc).  And configure rpc.period with the update interval.
>> These metrics don't show up as live counters in JMX, they're only pushed
>> out
>> to JMX during the periodic updates.
>>
>> If you've already configured all of those conditions and the metrics
>> _still_
>> aren't showing up, then let me know and I'll try to do some testing on
>> trunk
>> tomorrow.
>>
>> --gh
>>
>>
>> On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
>> <Kannan@facebook.com>wrote:
>>
>> > > they don't exist until used?
>> >
>> > I was doing a whole bunch of puts & gets against the cluster during that
>> > time. That's what you mean, correct?
>> >
>> > Regards,
>> > Kannan
>> > -----Original Message-----
>> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> > Sent: Wednesday, July 28, 2010 8:21 PM
>> > To: dev@hbase.apache.org
>> > Cc: hbase-dev@hadoop.apache.org
>> > Subject: Re: RPC statistics from region servers...
>> >
>> > I think those are dynamic metrics... they don't exist until used?
>> >
>> > On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Kannan@facebook.com>
>> > wrote:
>> > > I was using jconsole to check the stats published by region servers..
>> > >
>> > > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
>> > >
>> > > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
>> > > and,
>> > > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
>> > >
>> > > I no longer see attributes like:
>> > > "incrementColumnValueNumOps "
>> > > "incrementColumnValueAvgTime "
>> > > "putNumOps "
>> > > "putAvgTime "
>> > > "getNumOps "
>> > > "getAvgTime "
>> > > "deleteNumOps "
>> > > "deleteAvgTime "
>> > > "openScannerNumOps "
>> > > "openScannerAvtTime "
>> > > "nextNumOps "
>> > > "nextAvgTime "
>> > > "closeNumOps "
>> > > "closeAvgTime "
>> > >
>> > > Anyone aware of what change happened in this area?
>> > >
>> > > regards,
>> > > Kannan
>> > >
>> > >
>> >
>>
>

Mime
View raw message