hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yu Li <car...@gmail.com>
Subject Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Date Sat, 03 Jun 2017 08:32:07 GMT
HBASE-15160 is done with great efforts from @Enis. You could check the
latest patch there and do a customized backport if needed. @jeff

Best Regards,
Yu

On 16 March 2017 at 02:38, Enis Söztutar <enis.soz@gmail.com> wrote:

> If I remember from the jira, it got stuck on perf testing the different
> approaches of incrementing the histograms inline versus using a
> multi-producer single consumer queue (blockingQueue, or Disruptor, etc),
> and emitting the metrics inline and updating the histograms from a
> background thread. Which approach is better should be tested.
>
> Let's revive the jira.
> Enis
>
> On Tue, Mar 14, 2017 at 6:50 PM, Yu Li <carp84@gmail.com> wrote:
>
> > Let me check with @Enis in JIRA and get back to you (maybe days, due to
> > schedule) later.
> >
> > Best Regards,
> > Yu
> >
> > On 15 March 2017 at 05:57, jeff saremi <jeffsaremi@hotmail.com> wrote:
> >
> > > What's involved in getting this change merged into the main branch?
> These
> > > 2 counters (fsReadLatency, fsWriteLatency) are super important to us
> > > understanding what goes on behind every request. These are the minimum
> we
> > > need to have especially in the absence of HTrace.
> > >
> > > I just checked the latest code and it does not seem to have these.
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> > >
> > > ------------------------------
> > > *From:* jeff saremi <jeffsaremi@hotmail.com>
> > > *Sent:* Sunday, March 5, 2017 3:41:05 PM
> > > *To:* Yu Li; dev@hbase.apache.org
> > > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > >
> > > HI Yu
> > >
> > > I applied your path (the one that didn't have any V number) to 1.2.2
> > > manually. I'm waiting to see the results. I should circle back if there
> > are
> > > issues.
> > > ------------------------------
> > > *From:* Yu Li <carp84@gmail.com>
> > > *Sent:* Sunday, March 5, 2017 7:06:27 AM
> > > *To:* dev@hbase.apache.org
> > > *Cc:* jeffsaremi@hotmail.com
> > > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > @jeff normally it's recommended to try the latest one, say v5 in this
> > > case.
> > >
> > > btw, one thing to make clear: only #4 in my list is missing in current
> > > code base, all others are already there for usage.
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 4 March 2017 at 04:02, Sean Busbey <busbey@apache.org> wrote:
> > >
> > >> -user@hbase to bcc
> > >> +dev@hbase
> > >>
> > >> Jeff,
> > >>
> > >> These kinds of questions get into things that we only handle within
> > >> the development community. I've copied in the appropriate dev@hbase
> > >> mailing list, please ensure you are subscribed there so you can see
> > >> any further discussion.
> > >>
> > >> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <jeffsaremi@hotmail.com>
> > >> wrote:
> > >> > Yu
> > >> > Of the patches attached to HBASE-15160, do I need to apply all (v2,
> > v3,
> > >> ...) or just  HBASE-15160.patch ?
> > >> > Also how would I know against what version this patch was created?
> > >> > thanks
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > ________________________________
> > >> > From: jeff saremi <jeffsaremi@hotmail.com>
> > >> > Sent: Friday, March 3, 2017 10:34:00 AM
> > >> > To: Hbase-User
> > >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> > >> stage of processing a request
> > >> >
> > >> > Thanks a lot Yu
> > >> >
> > >> > These are truly the metrics we care about about at this point. It
is
> > >> sad to see that such important metrics were removed from the code.
> > >> >
> > >> > I will try to apply your patch on my own to the version of HBase we
> > >> have. We definitely need these.
> > >> >
> > >> > Other solutions like HTrace are not as urgent as having these few
> > >> metrics you talked about here. So if we can get these merged with the
> > code
> > >> we should be happy.
> > >> >
> > >> >
> > >> > ________________________________
> > >> > From: Yu Li <carp84@gmail.com>
> > >> > Sent: Friday, March 3, 2017 9:54:29 AM
> > >> > To: Hbase-User
> > >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> > >> stage of processing a request
> > >> >
> > >> > Hi Jeff,
> > >> >
> > >> > If the question is simply monitoring HDFS read/write latencies,
> please
> > >> > refer to HBASE-15160 <https://issues.apache.org/
> > jira/browse/HBASE-15160
> > >> >,
> > >> > there's a patch but not committed yet, and probably cannot apply
> > >> cleanly on
> > >> > current code base, but still some good reference IMHO, so JFYI.
> > >> >
> > >> > To get an overview of how quickly the system could respond and what
> > >> might
> > >> > be the root cause of the spikes, we only need to monitor the
> > >> > average/p99/p999 latency of below metrics (stages):
> > >> > 1. totalCallTime: time from request arriving at server to sending
> > >> response
> > >> > 2. processCallTime: time for the server to process the call,
> > regardless
> > >> of
> > >> > the time this call being queued
> > >> > 3. queueCallTime: time the call has been queued
> > >> > 4. HDFS read/pread/write time: time of HFile reading/writing, added
> in
> > >> > HBASE-15160
> > >> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
> > >> request
> > >> >
> > >> > However, for your original question, that to monitor the whole trace
> > of
> > >> a
> > >> > single request, I'm afraid no mature solution for the time being
> just
> > as
> > >> > Stack mentioned.
> > >> >
> > >> > Hope my answer helps (smile).
> > >> >
> > >> > Best Regards,
> > >> > Yu
> > >> >
> > >> > On 4 March 2017 at 00:48, jeff saremi <jeffsaremi@hotmail.com>
> wrote:
> > >> >
> > >> >> anything would help. thanks
> > >> >>
> > >> >> ________________________________
> > >> >> From: saint.ack@gmail.com <saint.ack@gmail.com> on behalf
of
> Stack <
> > >> >> stack@duboce.net>
> > >> >> Sent: Thursday, March 2, 2017 9:53:41 PM
> > >> >> To: Hbase-User
> > >> >> Subject: Re: Need guidance on getting detailed elapsed times in
> every
> > >> >> stage of processing a request
> > >> >>
> > >> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <
> jeffsaremi@hotmail.com
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > So i'd like to come back to my original question on how to
get
> > about
> > >> >> > separating the latency of HDFS from HBase.
> > >> >> >
> > >> >> >
> > >> >> That is a simple question to which we do not have an answer
> > >> unfortunately
> > >> >> (we should). If interested, I could describe how you might do
it. I
> > >> don't
> > >> >> think it would take much work.
> > >> >>
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >>
> > >> >> > Is there a most appropriate log4j TRACE option that could
print
> out
> > >> this
> > >> >> > information to the logs?
> > >> >> > Thanks
> > >> >> >
> > >> >> > ________________________________
> > >> >> > From: jeff saremi <jeffsaremi@hotmail.com>
> > >> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
> > >> >> > To: Hbase-User
> > >> >> > Subject: Re: Need guidance on getting detailed elapsed times
in
> > every
> > >> >> > stage of processing a request
> > >> >> >
> > >> >> > Thanks so much for the advice! Looking forward to when Tracing
> gets
> > >> >> picked
> > >> >> > up again
> > >> >> >
> > >> >> > ________________________________
> > >> >> > From: saint.ack@gmail.com <saint.ack@gmail.com> on
behalf of
> > Stack <
> > >> >> > stack@duboce.net>
> > >> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
> > >> >> > To: Hbase-User
> > >> >> > Subject: Re: Need guidance on getting detailed elapsed times
in
> > every
> > >> >> > stage of processing a request
> > >> >> >
> > >> >> > HBase/HTrace integration once worked but has long since rotted.
> > >> >> > Refactorings of internals without proper respect for trace
> > >> connections is
> > >> >> > the main culprit. Updates in htrace and hdfs that need attention
> > >> >> > reconnecting spans, etc., is another. On top of this, zipkin
> > project
> > >> has
> > >> >> > seen a burst of effort of late that would seem to offer much
> > promise
> > >> if
> > >> >> > someone of us spent some time rejiggering how HTrace and
Zipkin
> > >> relate.
> > >> >> >
> > >> >> > I would not waste any time on trying to setup HTrace for
HBase at
> > >> least
> > >> >> > until after HBASE-14451 goes in, an issue that has been put
aside
> > >> with a
> > >> >> > while now. Sorry if you've burned time on this to date.
> > >> >> >
> > >> >> > Yours,
> > >> >> > St.Ack
> > >> >> >
> > >> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <
> > jeffsaremi@hotmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Where would i seek help for issues revolving around
HTrace and
> > >> zipkin?
> > >> >> > > Here? Because I have configured everything the way
> documentation
> > >> said
> > >> >> > but i
> > >> >> > > see nothing in the zipkin server or in the logs. nothing
at all
> > >> >> > >
> > >> >> > > ________________________________
> > >> >> > > From: jeff saremi <jeffsaremi@hotmail.com>
> > >> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > >> >> > > To: user@hbase.apache.org
> > >> >> > > Subject: Re: Need guidance on getting detailed elapsed
times in
> > >> every
> > >> >> > > stage of processing a request
> > >> >> > >
> > >> >> > > No I had not. but it looks like what i needed. Thanks
Ted.
> > >> >> > >
> > >> >> > > I'll see if I have any more questions after reading
this.
> > >> >> > >
> > >> >> > > ________________________________
> > >> >> > > From: Ted Yu <yuzhihong@gmail.com>
> > >> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > >> >> > > To: user@hbase.apache.org
> > >> >> > > Subject: Re: Need guidance on getting detailed elapsed
times in
> > >> every
> > >> >> > > stage of processing a request
> > >> >> > >
> > >> >> > > Have you looked at:
> > >> >> > > http://hbase.apache.org/book.html#tracing
> > >> >> > >
> > >> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
> > >> jeffsaremi@hotmail.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > I think we need to get detailed information from
HBase
> > >> RegionServer
> > >> >> > logs
> > >> >> > > > on how a request (read or write) is processed.
Specifically
> > >> >> speaking, i
> > >> >> > > > need to know of say 100 ms time spent in processing
a write,
> > how
> > >> much
> > >> >> > of
> > >> >> > > it
> > >> >> > > > was spent waiting for the HDFS?
> > >> >> > > > What is the most efficient way of enabling this
in log4j
> > >> properties?
> > >> >> > Are
> > >> >> > > > there better mechanisms to get this information?
> > >> >> > > >
> > >> >> > > > If I can get this in the log, then I can process
the logs
> > >> offline or
> > >> >> in
> > >> >> > > > neartime and mount some dashboards on the top.
> > >> >> > > >
> > >> >> > > > thanks
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > Jeff
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message