hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeff saremi <jeffsar...@hotmail.com>
Subject Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Date Fri, 03 Mar 2017 19:13:28 GMT
Yu
Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) or just  HBASE-15160.patch
?
Also how would I know against what version this patch was created?
thanks




________________________________
From: jeff saremi <jeffsaremi@hotmail.com>
Sent: Friday, March 3, 2017 10:34:00 AM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing
a request

Thanks a lot Yu

These are truly the metrics we care about about at this point. It is sad to see that such
important metrics were removed from the code.

I will try to apply your patch on my own to the version of HBase we have. We definitely need
these.

Other solutions like HTrace are not as urgent as having these few metrics you talked about
here. So if we can get these merged with the code we should be happy.


________________________________
From: Yu Li <carp84@gmail.com>
Sent: Friday, March 3, 2017 9:54:29 AM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing
a request

Hi Jeff,

If the question is simply monitoring HDFS read/write latencies, please
refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
there's a patch but not committed yet, and probably cannot apply cleanly on
current code base, but still some good reference IMHO, so JFYI.

To get an overview of how quickly the system could respond and what might
be the root cause of the spikes, we only need to monitor the
average/p99/p999 latency of below metrics (stages):
1. totalCallTime: time from request arriving at server to sending response
2. processCallTime: time for the server to process the call, regardless of
the time this call being queued
3. queueCallTime: time the call has been queued
4. HDFS read/pread/write time: time of HFile reading/writing, added in
HBASE-15160
5. WAL sync time: time of WAL sync to HDFS, critical path of writing request

However, for your original question, that to monitor the whole trace of a
single request, I'm afraid no mature solution for the time being just as
Stack mentioned.

Hope my answer helps (smile).

Best Regards,
Yu

On 4 March 2017 at 00:48, jeff saremi <jeffsaremi@hotmail.com> wrote:

> anything would help. thanks
>
> ________________________________
> From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Thursday, March 2, 2017 9:53:41 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <jeffsaremi@hotmail.com>
> wrote:
>
> > So i'd like to come back to my original question on how to get about
> > separating the latency of HDFS from HBase.
> >
> >
> That is a simple question to which we do not have an answer unfortunately
> (we should). If interested, I could describe how you might do it. I don't
> think it would take much work.
>
> St.Ack
>
>
>
> > Is there a most appropriate log4j TRACE option that could print out this
> > information to the logs?
> > Thanks
> >
> > ________________________________
> > From: jeff saremi <jeffsaremi@hotmail.com>
> > Sent: Thursday, March 2, 2017 12:45:59 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > Thanks so much for the advice! Looking forward to when Tracing gets
> picked
> > up again
> >
> > ________________________________
> > From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf of Stack <
> > stack@duboce.net>
> > Sent: Thursday, March 2, 2017 12:17:35 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > HBase/HTrace integration once worked but has long since rotted.
> > Refactorings of internals without proper respect for trace connections is
> > the main culprit. Updates in htrace and hdfs that need attention
> > reconnecting spans, etc., is another. On top of this, zipkin project has
> > seen a burst of effort of late that would seem to offer much promise if
> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
> >
> > I would not waste any time on trying to setup HTrace for HBase at least
> > until after HBASE-14451 goes in, an issue that has been put aside with a
> > while now. Sorry if you've burned time on this to date.
> >
> > Yours,
> > St.Ack
> >
> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <jeffsaremi@hotmail.com>
> > wrote:
> >
> > > Where would i seek help for issues revolving around HTrace and zipkin?
> > > Here? Because I have configured everything the way documentation said
> > but i
> > > see nothing in the zipkin server or in the logs. nothing at all
> > >
> > > ________________________________
> > > From: jeff saremi <jeffsaremi@hotmail.com>
> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > No I had not. but it looks like what i needed. Thanks Ted.
> > >
> > > I'll see if I have any more questions after reading this.
> > >
> > > ________________________________
> > > From: Ted Yu <yuzhihong@gmail.com>
> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > Have you looked at:
> > > http://hbase.apache.org/book.html#tracing
> > >
> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <jeffsaremi@hotmail.com>
> > > wrote:
> > >
> > > > I think we need to get detailed information from HBase RegionServer
> > logs
> > > > on how a request (read or write) is processed. Specifically
> speaking, i
> > > > need to know of say 100 ms time spent in processing a write, how much
> > of
> > > it
> > > > was spent waiting for the HDFS?
> > > > What is the most efficient way of enabling this in log4j properties?
> > Are
> > > > there better mechanisms to get this information?
> > > >
> > > > If I can get this in the log, then I can process the logs offline or
> in
> > > > neartime and mount some dashboards on the top.
> > > >
> > > > thanks
> > > >
> > > >
> > > > Jeff
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message