hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yu Li <car...@gmail.com>
Subject Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Date Sun, 05 Mar 2017 15:06:27 GMT
@jeff normally it's recommended to try the latest one, say v5 in this case.

btw, one thing to make clear: only #4 in my list is missing in current code
base, all others are already there for usage.

Best Regards,
Yu

On 4 March 2017 at 04:02, Sean Busbey <busbey@apache.org> wrote:

> -user@hbase to bcc
> +dev@hbase
>
> Jeff,
>
> These kinds of questions get into things that we only handle within
> the development community. I've copied in the appropriate dev@hbase
> mailing list, please ensure you are subscribed there so you can see
> any further discussion.
>
> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <jeffsaremi@hotmail.com>
> wrote:
> > Yu
> > Of the patches attached to HBASE-15160, do I need to apply all (v2, v3,
> ...) or just  HBASE-15160.patch ?
> > Also how would I know against what version this patch was created?
> > thanks
> >
> >
> >
> >
> > ________________________________
> > From: jeff saremi <jeffsaremi@hotmail.com>
> > Sent: Friday, March 3, 2017 10:34:00 AM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
> >
> > Thanks a lot Yu
> >
> > These are truly the metrics we care about about at this point. It is sad
> to see that such important metrics were removed from the code.
> >
> > I will try to apply your patch on my own to the version of HBase we
> have. We definitely need these.
> >
> > Other solutions like HTrace are not as urgent as having these few
> metrics you talked about here. So if we can get these merged with the code
> we should be happy.
> >
> >
> > ________________________________
> > From: Yu Li <carp84@gmail.com>
> > Sent: Friday, March 3, 2017 9:54:29 AM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
> >
> > Hi Jeff,
> >
> > If the question is simply monitoring HDFS read/write latencies, please
> > refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160
> >,
> > there's a patch but not committed yet, and probably cannot apply cleanly
> on
> > current code base, but still some good reference IMHO, so JFYI.
> >
> > To get an overview of how quickly the system could respond and what might
> > be the root cause of the spikes, we only need to monitor the
> > average/p99/p999 latency of below metrics (stages):
> > 1. totalCallTime: time from request arriving at server to sending
> response
> > 2. processCallTime: time for the server to process the call, regardless
> of
> > the time this call being queued
> > 3. queueCallTime: time the call has been queued
> > 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> > HBASE-15160
> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
> request
> >
> > However, for your original question, that to monitor the whole trace of a
> > single request, I'm afraid no mature solution for the time being just as
> > Stack mentioned.
> >
> > Hope my answer helps (smile).
> >
> > Best Regards,
> > Yu
> >
> > On 4 March 2017 at 00:48, jeff saremi <jeffsaremi@hotmail.com> wrote:
> >
> >> anything would help. thanks
> >>
> >> ________________________________
> >> From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf of Stack <
> >> stack@duboce.net>
> >> Sent: Thursday, March 2, 2017 9:53:41 PM
> >> To: Hbase-User
> >> Subject: Re: Need guidance on getting detailed elapsed times in every
> >> stage of processing a request
> >>
> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <jeffsaremi@hotmail.com>
> >> wrote:
> >>
> >> > So i'd like to come back to my original question on how to get about
> >> > separating the latency of HDFS from HBase.
> >> >
> >> >
> >> That is a simple question to which we do not have an answer
> unfortunately
> >> (we should). If interested, I could describe how you might do it. I
> don't
> >> think it would take much work.
> >>
> >> St.Ack
> >>
> >>
> >>
> >> > Is there a most appropriate log4j TRACE option that could print out
> this
> >> > information to the logs?
> >> > Thanks
> >> >
> >> > ________________________________
> >> > From: jeff saremi <jeffsaremi@hotmail.com>
> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> > stage of processing a request
> >> >
> >> > Thanks so much for the advice! Looking forward to when Tracing gets
> >> picked
> >> > up again
> >> >
> >> > ________________________________
> >> > From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf of Stack
<
> >> > stack@duboce.net>
> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> > stage of processing a request
> >> >
> >> > HBase/HTrace integration once worked but has long since rotted.
> >> > Refactorings of internals without proper respect for trace
> connections is
> >> > the main culprit. Updates in htrace and hdfs that need attention
> >> > reconnecting spans, etc., is another. On top of this, zipkin project
> has
> >> > seen a burst of effort of late that would seem to offer much promise
> if
> >> > someone of us spent some time rejiggering how HTrace and Zipkin
> relate.
> >> >
> >> > I would not waste any time on trying to setup HTrace for HBase at
> least
> >> > until after HBASE-14451 goes in, an issue that has been put aside
> with a
> >> > while now. Sorry if you've burned time on this to date.
> >> >
> >> > Yours,
> >> > St.Ack
> >> >
> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <jeffsaremi@hotmail.com>
> >> > wrote:
> >> >
> >> > > Where would i seek help for issues revolving around HTrace and
> zipkin?
> >> > > Here? Because I have configured everything the way documentation
> said
> >> > but i
> >> > > see nothing in the zipkin server or in the logs. nothing at all
> >> > >
> >> > > ________________________________
> >> > > From: jeff saremi <jeffsaremi@hotmail.com>
> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> >> > > To: user@hbase.apache.org
> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> > > stage of processing a request
> >> > >
> >> > > No I had not. but it looks like what i needed. Thanks Ted.
> >> > >
> >> > > I'll see if I have any more questions after reading this.
> >> > >
> >> > > ________________________________
> >> > > From: Ted Yu <yuzhihong@gmail.com>
> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> >> > > To: user@hbase.apache.org
> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> > > stage of processing a request
> >> > >
> >> > > Have you looked at:
> >> > > http://hbase.apache.org/book.html#tracing
> >> > >
> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
> jeffsaremi@hotmail.com>
> >> > > wrote:
> >> > >
> >> > > > I think we need to get detailed information from HBase
> RegionServer
> >> > logs
> >> > > > on how a request (read or write) is processed. Specifically
> >> speaking, i
> >> > > > need to know of say 100 ms time spent in processing a write,
how
> much
> >> > of
> >> > > it
> >> > > > was spent waiting for the HDFS?
> >> > > > What is the most efficient way of enabling this in log4j
> properties?
> >> > Are
> >> > > > there better mechanisms to get this information?
> >> > > >
> >> > > > If I can get this in the log, then I can process the logs offline
> or
> >> in
> >> > > > neartime and mount some dashboards on the top.
> >> > > >
> >> > > > thanks
> >> > > >
> >> > > >
> >> > > > Jeff
> >> > > >
> >> > >
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message