Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C0298200C46 for ; Wed, 15 Mar 2017 02:11:08 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BEB3D160B89; Wed, 15 Mar 2017 01:11:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BAA3C160B7E for ; Wed, 15 Mar 2017 02:11:07 +0100 (CET) Received: (qmail 98352 invoked by uid 500); 15 Mar 2017 01:11:02 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 98341 invoked by uid 99); 15 Mar 2017 01:11:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Mar 2017 01:11:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 5A563C05A9 for ; Wed, 15 Mar 2017 01:11:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id s3JEtkAQA3RB for ; Wed, 15 Mar 2017 01:10:59 +0000 (UTC) Received: from mail-yw0-f179.google.com (mail-yw0-f179.google.com [209.85.161.179]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A18095F474 for ; Wed, 15 Mar 2017 01:10:59 +0000 (UTC) Received: by mail-yw0-f179.google.com with SMTP id v198so1737170ywc.2 for ; Tue, 14 Mar 2017 18:10:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=2C81al6EcczfkmF6nfmZkgH5sZjAqbzwAUJVx/ZOUns=; b=TIRcuw/Fos2Zf4SQq+DswoDQ5oXJ3083pVm7HmgZ9/JkfOa21/IVxEwYrFl24XZWtm P/cwYD7ClPkOImrSaN37O4A4rKw9TzKTqe+L7Cp1/D5RMdpWT88qPxaDiNGXRdhYWzRw YdzK1TN1HWd14WayvUyYBKvT7cFW/DKWADVbJ6CHhYa+b3gOgZOJKb9JAYvhbY/hwMnd +idkOtr/4yiQNINm8bUUgNawTYXu4GZcDVkhZCBkJ1hCMOQ3guhheVeMcexbe7zBHK47 X9b8PrkbX07mb1S6P6pe+kHKi5/OGWLGP/uFYTMAg9QvzqxVF9NoSiBsXxhDhSE4uYbW F5OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=2C81al6EcczfkmF6nfmZkgH5sZjAqbzwAUJVx/ZOUns=; b=maJlcbWl4UeZO53xqOs1NC8egkGhx2uRqBCZ8TUBkcWzOdJGfvgivfPlTv/wgjsLmf srccZUp4gR5wvk0yKx2KJILq8gSGwNeKWemyGFOooQsz/y8oqfopb2ziFkuhp/vKm8mf Habf5csYYWwPfmqZ9EQdg9FT0Qmw5/TSWCuVORps6ivrUXmo2YTKD5yHyGjDiB9q0GW0 Wy4kVeROfoZSkysh/DZOHytOQCQ116z3+2cl8PLX0rcxLTb7NrRjRoiO30DuByQeX1qM lFKkI/yfI0yn8Tzk4jbNgIogoYLY6+wcXZ1RRqGdxfXWe18/D0iZKkV5TGFuvonnz+Dc socA== X-Gm-Message-State: AFeK/H219ETMJrHqIw5zIl20ATh7DrawitcfE5T/zLStjKJKUvaFc2+DCAMmAql4J/alz06bGui3Fj09cLG71A== X-Received: by 10.129.157.83 with SMTP id u80mr435608ywg.323.1489540259216; Tue, 14 Mar 2017 18:10:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.125.86 with HTTP; Tue, 14 Mar 2017 18:10:58 -0700 (PDT) In-Reply-To: References: From: Jason Heo Date: Wed, 15 Mar 2017 10:10:58 +0900 Message-ID: Subject: Re: What does RowSet Compaction Duration means? To: user@kudu.apache.org Content-Type: multipart/alternative; boundary=94eb2c0b7320078562054aba9eb5 archived-at: Wed, 15 Mar 2017 01:11:08 -0000 --94eb2c0b7320078562054aba9eb5 Content-Type: text/plain; charset=UTF-8 Hi Alexey. Thank you for your reply. With your help, now I can understand what 'compact_rs_duration` means. But the `default_num_replicas` is just 3 not 5 :( It seems compaction on tableB affects huge on bulk loading on tableA. Is there a way to minimize compaction activities? (something like changing configuration of Kudu) The FAQ says that "Since compactions are so predictable, the only tuning knob available is the number of threads dedicated to flushes and compactions in the *maintenance manager*." my `maintenance_manager_num_threads` is already 1. Thanks. 2017-03-15 3:48 GMT+09:00 Alexey Serbin : > Hi Jason, > > As I understand, that 'milliseconds / second' cryptic unit means 'number > of units / for sampling (or averaging) interval'. > > I.e., they capture that metric reading (expressed in milliseconds) every > second, subtract previous value from the current value, and declare the > result as the result measurement at current time. If not capturing every > second, then it's about measuring every X seconds, do the subtraction of > the previous from the current measurement, and then divide by X. > > For a single tablet, the 'compact_rs_duration' metric stands for 'Time > spent compacting RowSets'. As I understand, that 'total_kudu_compact_rs_ > duration_sum_rate_across_kudu_replicas' is sum/accumulation of those > measurements for all existing replicas of the specified tablet across Kudu > cluster. > > I suspect you have the replication factor of 5 for that tablet, and at > some point all replicas become busy with rowset compaction all the time. > > Compactions on tables are run in the background. Compactions on different > tables run independently. So, if you have some other activity doing > inserts/updates on tableB, then it's natural to see compaction happen on > tabletB as well. > > > Best regards, > > Alexey > > On Tue, Mar 14, 2017 at 12:50 AM, Jason Heo > wrote: > >> Hi. >> >> I'm stuck with performance degradation on compaction happens. >> >> My Duration is "4956.71 milliseconds / second" What does this mean? I >> can't figure it out. >> >> Here is the captured image: http://imgur.com/WU9sRRq >> >> When I'm doing bulk indexing on tableA, sometimes compaction happens over >> tableB. Is this situation is natural? >> >> Thanks. >> > > --94eb2c0b7320078562054aba9eb5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Alexey.

Thank you for your reply.

With your help, now I can understand what 'compa= ct_rs_duration` means. But the `default_num_replicas` is just 3 not 5 :(

It seems compaction on tableB affects huge on bulk l= oading on tableA. Is there a way to minimize compaction activities? (someth= ing like changing configuration of Kudu)

The FAQ s= ays that "Since compactions = are so predictable, the only tuning knob available is the number of threads= dedicated to flushes and compactions in the=C2=A0maintenance manager."

my `maintenance_ma= nager_num_threads` is already 1.

= Thanks.

2017-03-15 3:48 GMT+09:00 A= lexey Serbin <aserbin@cloudera.com>:
Hi Jason,

As I under= stand, that 'milliseconds / second' cryptic unit means 'number = of units / for sampling (or averaging) interval'.

= I.e., they capture that metric reading (expressed in milliseconds) every se= cond, subtract previous value from the current value, and declare the resul= t as the result measurement at current time.=C2=A0 If not capturing every s= econd, then it's about measuring every X seconds, do the subtraction of= the previous from the current measurement, and then divide by X.
<= br>
For a single tablet, the 'compact_rs_duration' metric s= tands for 'Time spent compacting RowSets'.=C2=A0 As I understand, t= hat 'total_kudu_compact_rs_duration_sum_rate_across_kudu_repl= icas' is sum/accumulation of those measurements for all existing replic= as of the specified tablet across Kudu cluster.

I susp= ect you have the replication factor of 5 for that tablet, and at some point= all replicas become busy with rowset compaction all the time.

=
Compactions on tables are run in the background.=C2=A0 Compactions= on different tables run independently.=C2=A0 So, if you have some other ac= tivity doing inserts/updates on tableB, then it's natural to see compac= tion happen on tabletB as well.


Best re= gards,

Alexey

On Tue= , Mar 14, 2017 at 12:50 AM, Jason Heo <jason.heo.sde@gmail.com&g= t; wrote:
Hi.
I'm stuck with performance degradation on compaction ha= ppens.

My Duration is "4956.71 milliseconds /= second" What does this mean? I can't figure it out.
Here is the captured image:=C2=A0http://imgur.com/WU9sRRq

When I'm doing bulk indexing on tableA, sometimes compaction h= appens over tableB. Is this situation is natural?

= Thanks.


--94eb2c0b7320078562054aba9eb5--