Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BC6AE200B80 for ; Wed, 14 Sep 2016 16:04:29 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BB031160ABA; Wed, 14 Sep 2016 14:04:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B2C23160AB4 for ; Wed, 14 Sep 2016 16:04:28 +0200 (CEST) Received: (qmail 14634 invoked by uid 500); 14 Sep 2016 14:04:27 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 14624 invoked by uid 99); 14 Sep 2016 14:04:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Sep 2016 14:04:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6FF6A1A5C51 for ; Wed, 14 Sep 2016 14:04:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.899 X-Spam-Level: * X-Spam-Status: No, score=1.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 91jn1D4WVlS0 for ; Wed, 14 Sep 2016 14:04:24 +0000 (UTC) Received: from mail-yw0-f173.google.com (mail-yw0-f173.google.com [209.85.161.173]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 360C55FAD6 for ; Wed, 14 Sep 2016 14:04:23 +0000 (UTC) Received: by mail-yw0-f173.google.com with SMTP id i129so21449173ywb.0 for ; Wed, 14 Sep 2016 07:04:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=0JomutHujKdG7fwTSfeMwQVufbTomeHCHKFn3htDfVA=; b=D0M3A2MF1LUNSbaOhDzHvZMlJ99YA5E8UHcJjjOQWdYJhq2bNSINLIZT7YfP4b/U6E cKhXHkRArpmvSlbdY9q6XACkNHBTTt6WXyVP9x0GePZAlIsVcb2I9/ZagkbbrgN5VKe5 l2kxJtL/8f1yaGOh4BMiTqT5eNwnD/b5jPUI2UZXZswSLoljjiYyxWAt9QDTM+WLjLjb EELxqWcfcGYKqDR98s8s96pvpI0HKQ2Ma7l756baL2kma11ZYb2bq1jItcHC0ZbQiRJj O0YKioEQN4iTf8WHDub9ec7hLmygxJnEuQBXgpPPYyT+PoqwszY3EzMscSaHTxNSV/ti USMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=0JomutHujKdG7fwTSfeMwQVufbTomeHCHKFn3htDfVA=; b=HMBMBPGotaF+2C4IaLLpoN/zk5iqkG/75KtOFOf1YRGnhrjIf73Gpu5o29mFs22De/ 33rnjfPzW62v2GdO9Df12EQDkGm5NpKKOstQhwdJXgubT2n1vV8/NVTaWX3ICbcWLbt1 SVC0Mrg/80cMXtsZNdbRkjOoFrFY9DAhN9PPJK2OCjb+6iKCqa8jfgl4uBj5HAawU9vm WfWA4fJIIRCsW54iM4K/3dVU7RG6tNO0ldtqNa7Mmc7CXTml2BR8kEvnLjCnors1s/wn aRPGki2L26HPRP8t7Iczv45aabsNqB9SlnAcfe9Zm4W+69mI2Ru+jO+25/+AETyHmyr6 9F4Q== X-Gm-Message-State: AE9vXwMu9XUCP8IymFKVK7q6+DlYYCJXYMisWlbRV8kuFs8vcpzVfDxfCjUGQ4jOsCafKUp3d9pHCTS4AHAJnA== X-Received: by 10.129.49.204 with SMTP id x195mr2619102ywx.176.1473861862048; Wed, 14 Sep 2016 07:04:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.76.194 with HTTP; Wed, 14 Sep 2016 07:04:20 -0700 (PDT) Received: by 10.37.76.194 with HTTP; Wed, 14 Sep 2016 07:04:20 -0700 (PDT) In-Reply-To: References: <393687421.762328.1472044939599.JavaMail.zimbra@scai.fraunhofer.de> <2089159696.944715.1472627172729.JavaMail.zimbra@scai.fraunhofer.de> <57D482A8.7030507@gmail.com> <57D82FE4.8080103@gmail.com> <57D8C156.4070405@gmail.com> From: Josh Elser Date: Wed, 14 Sep 2016 10:04:20 -0400 Message-ID: Subject: Re: Accumulo Seek performance To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=001a11407234bc567d053c7834b9 archived-at: Wed, 14 Sep 2016 14:04:29 -0000 --001a11407234bc567d053c7834b9 Content-Type: text/plain; charset=UTF-8 Nope! My test harness (the github repo) doesn't show any noticeable difference between BatchScanner and Scanner. Would have to do more digging with Sven to figure out what's happening. One takeaway is lack of metrics to tell us what is actually happening is a major defect, imo. On Sep 14, 2016 9:53 AM, "Dylan Hutchison" wrote: > Do we have a (hopefully reproducible) conclusion from this thread, > regarding Scanners and BatchScanners? > > On Sep 13, 2016 11:17 PM, "Josh Elser" wrote: > >> Yeah, this seems to have been osx causing me grief. >> >> Spun up a 3tserver cluster (on openstack, even) and reran the same >> experiment. I could not reproduce the issues, even without substantial >> config tweaking. >> >> Josh Elser wrote: >> >>> I'm playing around with this a little more today and something is >>> definitely weird on my local machine. I'm seeing insane spikes in >>> performance using Scanners too. >>> >>> Coupled with Keith's inability to repro this, I am starting to think >>> that these are not worthwhile numbers to put weight behind. Something I >>> haven't been able to figure out is quite screwy for me. >>> >>> Josh Elser wrote: >>> >>>> Sven, et al: >>>> >>>> So, it would appear that I have been able to reproduce this one (better >>>> late than never, I guess...). tl;dr Serially using Scanners to do point >>>> lookups instead of a BatchScanner is ~20x faster. This sounds like a >>>> pretty serious performance issue to me. >>>> >>>> Here's a general outline for what I did. >>>> >>>> * Accumulo 1.8.0 >>>> * Created a table with 1M rows, each row with 10 columns using YCSB >>>> (workloada) >>>> * Split the table into 9 tablets >>>> * Computed the set of all rows in the table >>>> >>>> For a number of iterations: >>>> * Shuffle this set of rows >>>> * Choose the first N rows >>>> * Construct an equivalent set of Ranges from the set of Rows, choosing a >>>> random column (0-9) >>>> * Partition the N rows into X collections >>>> * Submit X tasks to query one partition of the N rows (to a thread pool >>>> with X fixed threads) >>>> >>>> I have two implementations of these tasks. One, where all ranges in a >>>> partition are executed via one BatchWriter. A second where each range is >>>> executed in serial using a Scanner. The numbers speak for themselves. >>>> >>>> ** BatchScanners ** >>>> 2016-09-10 17:51:38,811 [joshelser.YcsbBatchScanner] INFO : Shuffled all >>>> rows >>>> 2016-09-10 17:51:38,843 [joshelser.YcsbBatchScanner] INFO : All ranges >>>> calculated: 3000 ranges found >>>> 2016-09-10 17:51:38,846 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:52:19,025 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 40178 ms >>>> 2016-09-10 17:52:19,025 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:53:01,321 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 42296 ms >>>> 2016-09-10 17:53:01,321 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:53:47,414 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 46094 ms >>>> 2016-09-10 17:53:47,415 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:54:35,118 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 47704 ms >>>> 2016-09-10 17:54:35,119 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:55:24,339 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 49221 ms >>>> >>>> ** Scanners ** >>>> 2016-09-10 17:57:23,867 [joshelser.YcsbBatchScanner] INFO : Shuffled all >>>> rows >>>> 2016-09-10 17:57:23,898 [joshelser.YcsbBatchScanner] INFO : All ranges >>>> calculated: 3000 ranges found >>>> 2016-09-10 17:57:23,903 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:57:26,738 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 2833 ms >>>> 2016-09-10 17:57:26,738 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:57:29,275 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 2536 ms >>>> 2016-09-10 17:57:29,275 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:57:31,425 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 2150 ms >>>> 2016-09-10 17:57:31,425 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:57:33,487 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 2061 ms >>>> 2016-09-10 17:57:33,487 [joshelser.YcsbBatchScanner] INFO : Executing 6 >>>> range partitions using a pool of 6 threads >>>> 2016-09-10 17:57:35,628 [joshelser.YcsbBatchScanner] INFO : Queries >>>> executed in 2140 ms >>>> >>>> Query code is available >>>> https://github.com/joshelser/accumulo-range-binning >>>> >>>> Sven Hodapp wrote: >>>> >>>>> Hi Keith, >>>>> >>>>> I've tried it with 1, 2 or 10 threads. Unfortunately there where no >>>>> amazing differences. >>>>> Maybe it's a problem with the table structure? For example it may >>>>> happen that one row id (e.g. a sentence) has several thousand column >>>>> families. Can this affect the seek performance? >>>>> >>>>> So for my initial example it has about 3000 row ids to seek, which >>>>> will return about 500k entries. If I filter for specific column >>>>> families (e.g. a document without annotations) it will return about 5k >>>>> entries, but the seek time will only be halved. >>>>> Are there to much column families to seek it fast? >>>>> >>>>> Thanks! >>>>> >>>>> Regards, >>>>> Sven >>>>> >>>>> --001a11407234bc567d053c7834b9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Nope! My test harness (the github repo) doesn't show any= noticeable difference between BatchScanner and Scanner. Would have to do m= ore digging with Sven to figure out what's happening.

One takeaway is lack of metrics to tell us what is actually = happening is a major defect, imo.


On Sep 14, 2016 9= :53 AM, "Dylan Hutchison" <dhutchis@cs.washington.edu> wrote:

Do we have a (hopefully rep= roducible) conclusion from this thread, regarding Scanners and BatchScanner= s?


On Sep 13, 2016 1= 1:17 PM, "Josh Elser" <josh.elser@gmail.com> wrote:
Yeah, this seems to have been osx causin= g me grief.

Spun up a 3tserver cluster (on openstack, even) and reran the same experime= nt. I could not reproduce the issues, even without substantial config tweak= ing.

Josh Elser wrote:
I'm playing around with this a little more today and something is
definitely weird on my local machine. I'm seeing insane spikes in
performance using Scanners too.

Coupled with Keith's inability to repro this, I am starting to think that these are not worthwhile numbers to put weight behind. Something I
haven't been able to figure out is quite screwy for me.

Josh Elser wrote:
Sven, et al:

So, it would appear that I have been able to reproduce this one (better
late than never, I guess...). tl;dr Serially using Scanners to do point
lookups instead of a BatchScanner is ~20x faster. This sounds like a
pretty serious performance issue to me.

Here's a general outline for what I did.

* Accumulo 1.8.0
* Created a table with 1M rows, each row with 10 columns using YCSB
(workloada)
* Split the table into 9 tablets
* Computed the set of all rows in the table

For a number of iterations:
* Shuffle this set of rows
* Choose the first N rows
* Construct an equivalent set of Ranges from the set of Rows, choosing a random column (0-9)
* Partition the N rows into X collections
* Submit X tasks to query one partition of the N rows (to a thread pool
with X fixed threads)

I have two implementations of these tasks. One, where all ranges in a
partition are executed via one BatchWriter. A second where each range is executed in serial using a Scanner. The numbers speak for themselves.

** BatchScanners **
2016-09-10 17:51:38,811 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
2016-09-10 17:51:38,843 [joshelser.YcsbBatchScanner] INFO : All ranges
calculated: 3000 ranges found
2016-09-10 17:51:38,846 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:52:19,025 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 40178 ms
2016-09-10 17:52:19,025 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:53:01,321 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 42296 ms
2016-09-10 17:53:01,321 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:53:47,414 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 46094 ms
2016-09-10 17:53:47,415 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:54:35,118 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 47704 ms
2016-09-10 17:54:35,119 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:55:24,339 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 49221 ms

** Scanners **
2016-09-10 17:57:23,867 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
2016-09-10 17:57:23,898 [joshelser.YcsbBatchScanner] INFO : All ranges
calculated: 3000 ranges found
2016-09-10 17:57:23,903 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:57:26,738 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 2833 ms
2016-09-10 17:57:26,738 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:57:29,275 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 2536 ms
2016-09-10 17:57:29,275 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:57:31,425 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 2150 ms
2016-09-10 17:57:31,425 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:57:33,487 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 2061 ms
2016-09-10 17:57:33,487 [joshelser.YcsbBatchScanner] INFO : Executing 6
range partitions using a pool of 6 threads
2016-09-10 17:57:35,628 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 2140 ms

Query code is available
https://github.com/joshelser/accumulo-range-= binning

Sven Hodapp wrote:
Hi Keith,

I've tried it with 1, 2 or 10 threads. Unfortunately there where no
amazing differences.
Maybe it's a problem with the table structure? For example it may
happen that one row id (e.g. a sentence) has several thousand column
families. Can this affect the seek performance?

So for my initial example it has about 3000 row ids to seek, which
will return about 500k entries. If I filter for specific column
families (e.g. a document without annotations) it will return about 5k
entries, but the seek time will only be halved.
Are there to much column families to seek it fast?

Thanks!

Regards,
Sven

--001a11407234bc567d053c7834b9--