Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
From: jeff saremi <jeffsaremi@hotmail.com>
To: Yu Li <carp84@gmail.com>, "dev@hbase.apache.org" <dev@hbase.apache.org>
Subject: Re: Need guidance on getting detailed elapsed times in every stage of
 processing a request
Thread-Topic: Need guidance on getting detailed elapsed times in every stage
 of processing a request
Thread-Index: AQHSkgGUT46wS2IARUmdi07eUSr7FaF+46QAgAABXN6AAjL91IAA6A+AgAAHzOuAABvRRIAAfVmAgAC23rGAABKFgIAACoIqgAALD3aAAA5QAIAC0daAgACPcPY=
Date: Sun, 5 Mar 2017 23:41:05 +0000
Message-ID: <BY1PR14MB01671E6C602ABCBC32D1FC40C12D0@BY1PR14MB0167.namprd14.prod.outlook.com>
References: <BY1PR14MB0167F327B05FEF5B615209A9C1560@BY1PR14MB0167.namprd14.prod.outlook.com>
 <CALte62zaCWJrE1cKbqxNUJEo4BNvgYc8hPOL6s-VUQHPWf-fsw@mail.gmail.com>
 <BY1PR14MB0167D0EB07A89FE5D5ED5202C1560@BY1PR14MB0167.namprd14.prod.outlook.com>
 <BY1PR14MB01677E7107AF4A796192C9F0C1280@BY1PR14MB0167.namprd14.prod.outlook.com>
 <CADcMMgH6z+TiHhMQEKHmtQ9oY1KDBWLaQY-ygSCfp14Jxesofg@mail.gmail.com>
 <BY1PR14MB0167FB4269289BC146E39DBDC1280@BY1PR14MB0167.namprd14.prod.outlook.com>
 <BY1PR14MB01678DDAF06390DC7AD6A177C1280@BY1PR14MB0167.namprd14.prod.outlook.com>
 <CADcMMgFvMveyeqqanffyFMNLpdQDFUrKW9q5jAEYSwQbBOQ=4w@mail.gmail.com>
 <BY1PR14MB0167BD7ECD284E142E594BF9C12B0@BY1PR14MB0167.namprd14.prod.outlook.com>
 <CAM7-19JKh60cgH1uPqhg0QiBEa61=_dETxHhNs8zEDuO_6WNEQ@mail.gmail.com>
 <BY1PR14MB016712185BAD6764AD7DC412C12B0@BY1PR14MB0167.namprd14.prod.outlook.com>
 <BY1PR14MB0167EF1EC5C59F2A6BD65F23C12B0@BY1PR14MB0167.namprd14.prod.outlook.com>
 <CAN5cbe5UJRUY4jAeO=r8=MvFdFPXxvZn9uhzA2tx8n1END5V1g@mail.gmail.com>,<CAM7-19+bhtntJLzAit_qwNZH_SqaZ2yWbFrgU57-QxFJVmqAMQ@mail.gmail.com>
In-Reply-To: <CAM7-19+bhtntJLzAit_qwNZH_SqaZ2yWbFrgU57-QxFJVmqAMQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [TUVRa4uXmqG+G9LApjsQmbNmld088mpF]
x-incomingheadercount: 39
x-eopattributedmessage: 0
x-microsoft-exchange-diagnostics: 1;BN3NAM01HT034;5:LJLsSmy/kizdcE7SSc5Cwm4z6QhQsSMWYICrKk4iMnADENS335iT6yzDaXLGAkfYWUGTShYiRwoId/CNZV0URZQAN7F8Cw9Zr64EVtg0L32TBwO+f8a7SJOqgfpbK34pZLko8iNLchcnysClvmTzFA==;24:cB7T7aSyJGKl59ZqJkCUwj/d/w1H1az81kbiB6BhTCwVyY5zOZjCci4s9p54EPDvSg7z51H3Sdz3z3mdf51wmuSnn+KVmVa81yqJPLNES/s=;7:QkV+RZTAHz0ddOCF7TtBauRPXIxCt6YzrtCehiUU+tlDgpbSh2nQ/T+6H2ChpeGHW9eUe4ZRIvbE9xxE3fg6eE/9zuLvVh9YYBTBfnBjWzY72cUSSTdo4cXA/OkBZWv9YFIleqgr1m6EcUzPs0ECpgurNUhkvz8Kj/U9KhMM74e5XY9+gykni6GMiT4UZOiAW1i9OTxAYwsigezNAsfyvxzXbm/aHkq5Rf8tVuxnMAaYkweeYXktEXYb6sxIoSMooOfuEiQ1W7i9mV6bXBa5ULWDEHX3sCUyKckpNBkNKG+U0HSWH2qWlvjHjH1Idl0K
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900016);DIR:OUT;SFP:1102;SCL:1;SRVR:BN3NAM01HT034;H:BY1PR14MB0167.namprd14.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: df183f1c-0987-4885-52f5-08d464211767
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(1601125254)(1603101448)(1701031045);SRVR:BN3NAM01HT034;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015087)(444000031);SRVR:BN3NAM01HT034;BCL:0;PCL:0;RULEID:;SRVR:BN3NAM01HT034;
x-forefront-prvs: 02379661A3
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BY1PR14MB01671E6C602ABCBC32D1FC40C12D0BY1PR14MB0167namp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Mar 2017 23:41:05.4178
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3NAM01HT034
X-OriginalArrivalTime: 05 Mar 2017 23:41:07.0548 (UTC) FILETIME=[F62249C0:01D29609]
archived-at: Sun, 05 Mar 2017 23:41:23 -0000

--_000_BY1PR14MB01671E6C602ABCBC32D1FC40C12D0BY1PR14MB0167namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

HI Yu

I applied your path (the one that didn't have any V number) to 1.2.2 manual=
ly. I'm waiting to see the results. I should circle back if there are issue=
s.

________________________________
From: Yu Li <carp84@gmail.com>
Sent: Sunday, March 5, 2017 7:06:27 AM
To: dev@hbase.apache.org
Cc: jeffsaremi@hotmail.com
Subject: Re: Need guidance on getting detailed elapsed times in every stage=
 of processing a request

@jeff normally it's recommended to try the latest one, say v5 in this case.

btw, one thing to make clear: only #4 in my list is missing in current code=
 base, all others are already there for usage.

Best Regards,
Yu

On 4 March 2017 at 04:02, Sean Busbey <busbey@apache.org<mailto:busbey@apac=
he.org>> wrote:
-user@hbase to bcc
+dev@hbase

Jeff,

These kinds of questions get into things that we only handle within
the development community. I've copied in the appropriate dev@hbase
mailing list, please ensure you are subscribed there so you can see
any further discussion.

On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <jeffsaremi@hotmail.com<mailto:=
jeffsaremi@hotmail.com>> wrote:
> Yu
> Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, .=
..) or just  HBASE-15160.patch ?
> Also how would I know against what version this patch was created?
> thanks
>
>
>
>
> ________________________________
> From: jeff saremi <jeffsaremi@hotmail.com<mailto:jeffsaremi@hotmail.com>>
> Sent: Friday, March 3, 2017 10:34:00 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every sta=
ge of processing a request
>
> Thanks a lot Yu
>
> These are truly the metrics we care about about at this point. It is sad =
to see that such important metrics were removed from the code.
>
> I will try to apply your patch on my own to the version of HBase we have.=
 We definitely need these.
>
> Other solutions like HTrace are not as urgent as having these few metrics=
 you talked about here. So if we can get these merged with the code we shou=
ld be happy.
>
>
> ________________________________
> From: Yu Li <carp84@gmail.com<mailto:carp84@gmail.com>>
> Sent: Friday, March 3, 2017 9:54:29 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every sta=
ge of processing a request
>
> Hi Jeff,
>
> If the question is simply monitoring HDFS read/write latencies, please
> refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
> there's a patch but not committed yet, and probably cannot apply cleanly =
on
> current code base, but still some good reference IMHO, so JFYI.
>
> To get an overview of how quickly the system could respond and what might
> be the root cause of the spikes, we only need to monitor the
> average/p99/p999 latency of below metrics (stages):
> 1. totalCallTime: time from request arriving at server to sending respons=
e
> 2. processCallTime: time for the server to process the call, regardless o=
f
> the time this call being queued
> 3. queueCallTime: time the call has been queued
> 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> HBASE-15160
> 5. WAL sync time: time of WAL sync to HDFS, critical path of writing requ=
est
>
> However, for your original question, that to monitor the whole trace of a
> single request, I'm afraid no mature solution for the time being just as
> Stack mentioned.
>
> Hope my answer helps (smile).
>
> Best Regards,
> Yu
>
> On 4 March 2017 at 00:48, jeff saremi <jeffsaremi@hotmail.com<mailto:jeff=
saremi@hotmail.com>> wrote:
>
>> anything would help. thanks
>>
>> ________________________________
>> From: saint.ack@gmail.com<mailto:saint.ack@gmail.com> <saint.ack@gmail.c=
om<mailto:saint.ack@gmail.com>> on behalf of Stack <
>> stack@duboce.net<mailto:stack@duboce.net>>
>> Sent: Thursday, March 2, 2017 9:53:41 PM
>> To: Hbase-User
>> Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>>
>> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <jeffsaremi@hotmail.com<mai=
lto:jeffsaremi@hotmail.com>>
>> wrote:
>>
>> > So i'd like to come back to my original question on how to get about
>> > separating the latency of HDFS from HBase.
>> >
>> >
>> That is a simple question to which we do not have an answer unfortunatel=
y
>> (we should). If interested, I could describe how you might do it. I don'=
t
>> think it would take much work.
>>
>> St.Ack
>>
>>
>>
>> > Is there a most appropriate log4j TRACE option that could print out th=
is
>> > information to the logs?
>> > Thanks
>> >
>> > ________________________________
>> > From: jeff saremi <jeffsaremi@hotmail.com<mailto:jeffsaremi@hotmail.co=
m>>
>> > Sent: Thursday, March 2, 2017 12:45:59 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > Thanks so much for the advice! Looking forward to when Tracing gets
>> picked
>> > up again
>> >
>> > ________________________________
>> > From: saint.ack@gmail.com<mailto:saint.ack@gmail.com> <saint.ack@gmail=
.com<mailto:saint.ack@gmail.com>> on behalf of Stack <
>> > stack@duboce.net<mailto:stack@duboce.net>>
>> > Sent: Thursday, March 2, 2017 12:17:35 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > HBase/HTrace integration once worked but has long since rotted.
>> > Refactorings of internals without proper respect for trace connections=
 is
>> > the main culprit. Updates in htrace and hdfs that need attention
>> > reconnecting spans, etc., is another. On top of this, zipkin project h=
as
>> > seen a burst of effort of late that would seem to offer much promise i=
f
>> > someone of us spent some time rejiggering how HTrace and Zipkin relate=
.
>> >
>> > I would not waste any time on trying to setup HTrace for HBase at leas=
t
>> > until after HBASE-14451 goes in, an issue that has been put aside with=
 a
>> > while now. Sorry if you've burned time on this to date.
>> >
>> > Yours,
>> > St.Ack
>> >
>> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <jeffsaremi@hotmail.com<ma=
ilto:jeffsaremi@hotmail.com>>
>> > wrote:
>> >
>> > > Where would i seek help for issues revolving around HTrace and zipki=
n?
>> > > Here? Because I have configured everything the way documentation sai=
d
>> > but i
>> > > see nothing in the zipkin server or in the logs. nothing at all
>> > >
>> > > ________________________________
>> > > From: jeff saremi <jeffsaremi@hotmail.com<mailto:jeffsaremi@hotmail.=
com>>
>> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
>> > > To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>> > > Subject: Re: Need guidance on getting detailed elapsed times in ever=
y
>> > > stage of processing a request
>> > >
>> > > No I had not. but it looks like what i needed. Thanks Ted.
>> > >
>> > > I'll see if I have any more questions after reading this.
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>
>> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
>> > > To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>> > > Subject: Re: Need guidance on getting detailed elapsed times in ever=
y
>> > > stage of processing a request
>> > >
>> > > Have you looked at:
>> > > http://hbase.apache.org/book.html#tracing
>> > >
>> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <jeffsaremi@hotmail.co=
m<mailto:jeffsaremi@hotmail.com>>
>> > > wrote:
>> > >
>> > > > I think we need to get detailed information from HBase RegionServe=
r
>> > logs
>> > > > on how a request (read or write) is processed. Specifically
>> speaking, i
>> > > > need to know of say 100 ms time spent in processing a write, how m=
uch
>> > of
>> > > it
>> > > > was spent waiting for the HDFS?
>> > > > What is the most efficient way of enabling this in log4j propertie=
s?
>> > Are
>> > > > there better mechanisms to get this information?
>> > > >
>> > > > If I can get this in the log, then I can process the logs offline =
or
>> in
>> > > > neartime and mount some dashboards on the top.
>> > > >
>> > > > thanks
>> > > >
>> > > >
>> > > > Jeff
>> > > >
>> > >
>> >
>>


--_000_BY1PR14MB01671E6C602ABCBC32D1FC40C12D0BY1PR14MB0167namp_--