Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8F66917B5C for ; Thu, 8 Oct 2015 00:38:17 +0000 (UTC) Received: (qmail 64180 invoked by uid 500); 8 Oct 2015 00:38:14 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 64075 invoked by uid 500); 8 Oct 2015 00:38:13 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 64065 invoked by uid 99); 8 Oct 2015 00:38:13 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2015 00:38:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 675A3C4693 for ; Thu, 8 Oct 2015 00:38:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.885 X-Spam-Level: **** X-Spam-Status: No, score=4.885 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_IMAGE_RATIO_02=0.805, HTML_MESSAGE=3, KAM_LINEPADDING=1.2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ERbX_nbM-b1M for ; Thu, 8 Oct 2015 00:38:08 +0000 (UTC) Received: from mail-yk0-f169.google.com (mail-yk0-f169.google.com [209.85.160.169]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 71FB8204D9 for ; Thu, 8 Oct 2015 00:38:07 +0000 (UTC) Received: by ykft14 with SMTP id t14so35022329ykf.0 for ; Wed, 07 Oct 2015 17:38:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=WwZjcH5EM6ovzKCvEX+iBP6GHQL56FXaCQrBqSZ3tuE=; b=L7cO419AOMmUfpKcLWGM0KSw0Q5yTkY6JW9v6m8b0BDNp5QH7DvgYUFwTeRu8M2eYO CRCnZ3Cy69ojhNXPzeIcLIgduiFXGshpbu8t7VwQBXELWN4qDX9inT8YyEPDJOs8gCxn CbmnG/6L5LuiGTnEECoj/AAegqH2zbBVsYOosc3TSh9aqUeiBUDjKl/QCYi/TqfBw3Ze GnB0AtGuxbaW28A/fc6XgCnCe9YxTHwZcsz6rYye96QnfdH1wicGhOMCOCPoWqRVn2Ur PxCSLEMvC+jV8Uax6txC85g3kJROZFek+c5rGL0NQ3itZFx1CHfKXvhJzCmgGQUgRhNK xq/Q== MIME-Version: 1.0 X-Received: by 10.13.236.208 with SMTP id v199mr3322859ywe.20.1444264686344; Wed, 07 Oct 2015 17:38:06 -0700 (PDT) Received: by 10.37.210.197 with HTTP; Wed, 7 Oct 2015 17:38:06 -0700 (PDT) In-Reply-To: <83D73D519B6DA448976A53C2F5EB325DF9C3B061C1@MAIL01.corp.lgcns.com> References: <83D73D519B6DA448976A53C2F5EB325DF9C3B061C1@MAIL01.corp.lgcns.com> Date: Wed, 7 Oct 2015 17:38:06 -0700 Message-ID: Subject: Re: Asking about the trend of increasing latency, hbase spikes. From: Ted Yu To: user Content-Type: multipart/alternative; boundary=94eb2c0870dc97603905218d1327 --94eb2c0870dc97603905218d1327 Content-Type: text/plain; charset=UTF-8 This question should be directed to user@ Can you use third party site for the images - they didn't go through. On Wed, Oct 7, 2015 at 5:35 PM, UI-JIN LIM wrote: > Hi. This is Ui Jin, Lim in Korea, LG CNS > > > > We had setup and are operating hbase 0.98.13 on our customer, iptv > Company. > > Our customer claim about this spike. Following is our status. > > > > . We had been operating with BlockCache(heap) for 3 months. The ratio of > above 100ms has been reached 6.69% of total transaction. > > The following graph show the trend of executions, above 100ms per > minute, at June 17th. (This is the response time of 2~3 Hbase > transactions) > > [image: cid:image001.jpg@01D1011A.5E93FF70] > > > > - After Hbase restarting at June 18th, The spikes had been reduced > dramatically. But its ratio had been increasing continuously. > > [image: cid:image002.png@01D1011A.5E93FF70] > > > > - We had adopted off-heap bucket cache at August 19th. But the increasing > does not disappear. > > [image: cid:image002.png@01D10103.82E488C0] > > [image: cid:image002.png@01D10103.82E488C0] > > > > - We set HBase to log slow query which have above 10ms execution time. All > RegionServer logs show increasing its counts. > > > > [image: cid:image005.png@01D1011A.5E93FF70] > > > > > > What we concern is not the spike itself but its incremental trend. > > There is no increment of transactions and dataset. > > I ask your thoughts about this issue. > > > > > > > > > > ------------------------------------------------------------- > > *UI JIN, LIM* > > *DNA 3.0** - **LG CNS* > > Data Architecture Advisory > > Big Data Technical Support Team > > Mobile : +82) 010-8307-1210 > > > --94eb2c0870dc97603905218d1327 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
This question should be directed to user@

Can you use third party site for the images - they didn't go through= .

On W= ed, Oct 7, 2015 at 5:35 PM, UI-JIN LIM <ujlim@lgcns.com> wrote= :

Hi. This is Ui Jin, Lim in Korea, LG CNS

=C2=A0

We had setup and are operating hbase = 0.98.13 on our customer, iptv Company.

Our custom= er claim about this spike. Following is our status.

=C2=A0

. We had been operating with BlockCache(h= eap) for 3 months. The ratio of above 100ms has been reached 6.69% of total= transaction.

=C2=A0 The following graph show the = trend of executions, above 100ms per minute, at June 17th. (This= is the response time of 2~3 Hbase transactions)

3D"cid:image001.jpg=

=C2=A0=

- After Hbase restarting at June 18th, The spikes had been = reduced dramatically. But its ratio had been increasing continuously.

<= span lang=3D"EN-US">3D"cid:image002.=

=C2=A0

- We had adopted off-heap bucket cache at August 19th. But= the increasing does not disappear.

3D"cid:image002.png@01D10103.82E488C0"

3D"cid:image002.png@01D10103.82E488C0"

=C2=A0

- = We set HBase to log slow query which have above 10ms execution time. All Re= gionServer logs show increasing its counts.

=C2=A0

3D"cid:image005.png@01D1011A.5E93FF70"

=C2=A0

=C2=A0

What we concern is not the spike itself = but its incremental trend.

There is no increme= nt of transactions and dataset.

I ask your thou= ghts about this issue.

=C2=A0

=C2=A0

=C2=A0

=C2=A0

-----------------------------= --------------------------------

UI JIN, LIM=

DNA 3.0 - LG CNS

Data Architecture Advisory

<= p class=3D"MsoNormal" style=3D"layout-grid-mode:char">Big Data Technical Support Team

Mobile : +82) 010-8307-1210=C2=A0

=C2=A0


--94eb2c0870dc97603905218d1327--