Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B693D2009F9 for ; Mon, 23 May 2016 18:11:21 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B55DB160A0E; Mon, 23 May 2016 16:11:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 86EDF160A05 for ; Mon, 23 May 2016 18:11:20 +0200 (CEST) Received: (qmail 77361 invoked by uid 500); 23 May 2016 16:11:19 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 77351 invoked by uid 99); 23 May 2016 16:11:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2016 16:11:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 41DDDC062C for ; Mon, 23 May 2016 16:11:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.28 X-Spam-Level: * X-Spam-Status: No, score=1.28 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=deenlo-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id sbnhJm6dvQYe for ; Mon, 23 May 2016 16:11:15 +0000 (UTC) Received: from mail-oi0-f46.google.com (mail-oi0-f46.google.com [209.85.218.46]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 6A5C25FCAD for ; Mon, 23 May 2016 16:11:15 +0000 (UTC) Received: by mail-oi0-f46.google.com with SMTP id b65so140909340oia.1 for ; Mon, 23 May 2016 09:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=deenlo-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=d+S8CvnOyqc4v8nOy4nxUy4yjNBH7oaPsr3nnYR0jJw=; b=cB6pzmMiG3iVWHZgegAblMJ92U9mdQQALRUZcCNxOmtWNVaFWTQJmPG0C5mDLy0vUf jjCuUn2me7XrAGrNo/sOfhadpEifsaW1tJXkGxh6YYwYTWQrxbptIu9OgM/7JD5CTf88 n0fNqTL2k6MUFg3l7gLZOlT6H2IUk9qcc3i2hrbxgbtcbiq2JHSgHFE5x5beKM6SYPBb 7ExbsoU+Mir+MfYvRHi83frDLDzGdL4nSjp6rrwvSKMa94+XP3yJ9hrjyJgHIGAW5ouv HQRrUJak+WwZRdKAdqlAzuUdrYQ4r6fm28eNA1WQWTSMnDoRJyF7GYvwQNcOTgbFET1b Np1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=d+S8CvnOyqc4v8nOy4nxUy4yjNBH7oaPsr3nnYR0jJw=; b=h2KL3Zc+9DTjFW7gbj/zso12j1it/pnLGXGtno7CRQdxVvCC1m54u7XwFgdlACE6U0 vp29WeCDTHkvixXX+224UPvzmPnSzJ7MUxyVIybAcHOf4MqI82iHbkvhkyP/qfP/ExJL fY3FHjyCZj6k3idkqW2cf5m+Sa0WkJ7Jp8y+foKD7o+p9kZ3MsReXx645mpcTFan4RNN EYndClD7SCSVKjME1uW+qnyHwS/pxx++CrVfcWF4zctCyduChuklfPQ9wKg62XzpKzNE 5XgWXaR0LgEZWh3uvqux9JQ+nf+R9ph/b3Bf3YW3E9IqNeL0KVxosCMWniH87kTUNPtN XW3g== X-Gm-Message-State: ALyK8tK3NW/tAfYj3a0uYrP9gzqFsk+dEylmKF88rDD0+BFt1t26N5H3HnbVEUOwRsOGs17UJxsQZBY84ipjLQ== MIME-Version: 1.0 X-Received: by 10.202.68.138 with SMTP id r132mr4261544oia.72.1464019874284; Mon, 23 May 2016 09:11:14 -0700 (PDT) Received: by 10.202.216.6 with HTTP; Mon, 23 May 2016 09:11:14 -0700 (PDT) In-Reply-To: References: Date: Mon, 23 May 2016 12:11:14 -0400 Message-ID: Subject: Re: Feedback about techniques for tuning batch scanning for my problem From: Keith Turner To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=001a113dafd68d2e26053384b09b archived-at: Mon, 23 May 2016 16:11:21 -0000 --001a113dafd68d2e26053384b09b Content-Type: text/plain; charset=UTF-8 Until ACCUMULO-4164[1] is released you may see a performance difference between 1 and 2 rfile level indexes. However this depends on your #of seeks to scan ratio. If you seek to a spot and read a good bit of data from there, it probably will not matter. Accumulo caches a certain number of open rfiles[2]. For these cached open rfiles the root level node is kept in memory. However, levels of the index below the root are always read from index cache. This is where the change in ACCUMULO-4164 to avoid a copy from index cache helps. So you may want to consider ensuring the caching of open rfile is large enough and that the index depth is one (for now). [1]: https://issues.apache.org/jira/browse/ACCUMULO-4164 [2]: http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_scan_files_open_max On Mon, May 23, 2016 at 11:48 AM, Mario Pastorelli < mario.pastorelli@teralytics.ch> wrote: > This is interesting and I think that's the main parameters that I need to > tune. We have a lot of memory to spare for Accumulo so the index should fix > in memory, but I'll check when I change those parameters. With small > records, I think it makes sense to have smaller blocks for faster lookups. > Thank you! > > On Mon, May 23, 2016 at 5:16 PM, Keith Turner wrote: > >> Mario >> >> You could experiment with adjusting table.file.compress.blocksize[1] and >> table.file.compress.blocksize.index[2]. Making the data block size smaller >> will increase the files index size and may lower random seek times >> depending on how much of the index fits in memory. Adjusting the index >> block size will control the depth of the index tree. You can adjust these >> settings, compact, and run "accumulo rfile-info" on a file to see >> information about the index size, depth, and number of data blocks. You >> can also adjust the index[6] and data[5] cache size settings and turn >> data[3] and index[4] caching on or off for your table. >> >> [1]: >> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_file_compress_blocksize >> [2]: >> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_file_compress_blocksize_index >> [3]: >> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_cache_block_enable >> [4]: >> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_cache_index_enable >> [5]: >> http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_cache_data_size >> [6]: >> http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_cache_index_size >> >> Keith >> >> On Thu, May 19, 2016 at 11:08 AM, Mario Pastorelli < >> mario.pastorelli@teralytics.ch> wrote: >> >>> Hey people, >>> I'm trying to tune a bit the query performance to see how fast it can go >>> and I thought it would be great to have comments from the community. The >>> problem that I'm trying to solve in Accumulo is the following: we want to >>> store the entities that have been in a certain location in a certain day. >>> The location is a Long and the entity id is a Long. I want to be able to >>> scan ~1M of rows in few seconds, possibly less than one. Right now, I'm >>> doing the following things: >>> >>> 1. I'm using a sharding byte at the start of the rowId to keep the >>> data in the same range distributed in the cluster >>> 2. all the records are encoded, one single record is composed by >>> 1. rowId: 1 shard byte + 3 bytes for the day >>> 2. column family: 8 byte for the long corresponding to the hash >>> of the location >>> 3. column qualifier: 8 byte corresponding to the identifier of >>> the entity >>> 4. value: 2 bytes for some additional information >>> 3. I use a batch scanner because I don't need sorting and it's faster >>> >>> As expected, it takes few seconds to scan 1M rows but now I'm wondering >>> if I can improve it. My ideas are the following: >>> >>> 1. set table.compaction.major.ration to 1 because I don't care about >>> the ingestion performance and this should improve the query performance >>> 2. pre-split tables to match the number of servers and then use a >>> byte of shard as first byte of the rowId. This should improve both writing >>> and reading the data because both should work in parallel for what I >>> understood >>> 3. enable bloom filter on the table >>> >>> Do you think those ideas make sense? Furthermore, I have two questions: >>> >>> 1. considering that a single entry is only 22 bytes but I'm going to >>> scan ~1M records per query, do you think I should change the BatchScanner >>> buffers somehow? >>> 2. anything else to improve the scan speed? Again, I don't care >>> about the ingestion time >>> >>> Thanks for the help! >>> >>> -- >>> Mario Pastorelli | TERALYTICS >>> >>> *software engineer* >>> >>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland >>> phone: +41794381682 >>> email: mario.pastorelli@teralytics.ch >>> www.teralytics.net >>> >>> Company registration number: CH-020.3.037.709-7 | Trade register Canton >>> Zurich >>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, >>> Yann de Vries >>> >>> This e-mail message contains confidential information which is for the >>> sole attention and use of the intended recipient. Please notify us at once >>> if you think that it may not be intended for you and delete it immediately. >>> >> >> > > > -- > Mario Pastorelli | TERALYTICS > > *software engineer* > > Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland > phone: +41794381682 > email: mario.pastorelli@teralytics.ch > www.teralytics.net > > Company registration number: CH-020.3.037.709-7 | Trade register Canton > Zurich > Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann > de Vries > > This e-mail message contains confidential information which is for the > sole attention and use of the intended recipient. Please notify us at once > if you think that it may not be intended for you and delete it immediately. > --001a113dafd68d2e26053384b09b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Until ACCUMULO-4164[1] is released you may see a performan= ce difference between 1 and 2 rfile level indexes.=C2=A0 However this depen= ds on your #of seeks to scan ratio.=C2=A0 If you seek to a spot and read a = good bit of data from there, it probably will not matter.

Accumulo caches a certain number of open rfiles[2].=C2=A0 For these c= ached open rfiles the root level node is kept in memory. However, levels of= the index below the root are always read from index cache.=C2=A0 This is w= here the change in ACCUMULO-4164 to avoid a copy from index cache helps.=C2= =A0 So you may want to consider ensuring the caching of open rfile is large= enough and that the index depth is one (for now). =C2=A0=C2=A0
<= div>
[1]: https://issues.apache.org/jira/browse/ACCUMULO-4164
[2]: http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_sca= n_files_open_max

On Mon, May 23, 2016 at 11:48 AM, Mario Pastorelli <mario.pastorelli@teralytics.ch> wrote:
This is interesting and I think th= at's the main parameters that I need to tune. We have a lot of memory t= o spare for Accumulo so the index should fix in memory, but I'll check = when I change those parameters. With small records, I think it makes sense = to have smaller blocks for faster lookups. Thank you!

On Mon, May 23, 2016 at 5:16 PM, Keith Turner <keith@deenlo.co= m> wrote:
= Mario

You could experiment with adjusting table.file.compress.b= locksize[1] and=C2=A0 table.file.compress.blocksize.index[2].=C2=A0 Making = the data block size smaller will increase the files index size and may lowe= r random seek times depending on how much of the index fits in memory.=C2= =A0 Adjusting the index block size will control the depth of the index tree= .=C2=A0=C2=A0 You can adjust these settings, compact, and run "accumul= o rfile-info" on a file to see information about the index size, depth= , and number of data blocks.=C2=A0 You can also adjust the index[6] and dat= a[5] cache size settings and turn data[3] and index[4] caching on or off fo= r your table.

[1]: http://accu= mulo.apache.org/1.7/accumulo_user_manual#_table_file_compress_blocksize=
[2]: http://accumulo.apache= .org/1.7/accumulo_user_manual#_table_file_compress_blocksize_index
[= 3]: http://accumulo.apache.org/1.7/accumul= o_user_manual#_table_cache_block_enable
[4]: http://accumulo.apache.org/1.7/accumulo_user_manual#_table_cach= e_index_enable
[5]: http://accumulo.= apache.org/1.7/accumulo_user_manual#_tserver_cache_data_size
[6]: http://accumulo.apache.org/1.7/accumulo_user= _manual#_tserver_cache_index_size

Keith
=

On Thu, May 19, 2016 at 11:08 AM, Mario Pastorelli <ma= rio.pastorelli@teralytics.ch> wrote:
Hey people,
I'm t= rying to tune a bit the query performance to see how fast it can go and I t= hought it would be great to have comments from the community. The problem t= hat I'm trying to solve in Accumulo is the following: we want to store = the entities that have been in a certain location in a certain day. The loc= ation is a Long and the entity id is a Long. I want to be able to scan ~1M = of rows in few seconds, possibly less than one. Right now, I'm doing th= e following things:
  1. I'm using a sharding byte at the start o= f the rowId to keep the data in the same range distributed in the cluster
  2. all the records are encoded, one single record is composed by
  3. rowId: 1 shard byte + 3 bytes for the day
  4. column family: 8 by= te for the long corresponding to the hash of the location
  5. column qu= alifier: 8 byte corresponding to the identifier of the entity
  6. value= : 2 bytes for some additional information
  • I use a batch scanne= r because I don't need sorting and it's faster

    • As expec= ted, it takes few seconds to scan 1M rows but now I'm wondering if I ca= n improve it. My ideas are the following:

    1. set table.co= mpaction.major.ration to 1 because I don't care about the ingestion per= formance and this should improve the query performance
    2. pre-spli= t tables to match the number of servers and then use a byte of shard as fir= st byte of the rowId. This should improve both writing and reading the data= because both should work in parallel for what I understood
    3. enable = bloom filter on the table
    Do you think those ideas m= ake sense? Furthermore, I have two questions:
    1. considering that a= single entry is only 22 bytes but I'm going to scan ~1M records per qu= ery, do you think I should change the BatchScanner buffers somehow?
      2. anything else to improve the scan speed? Again, I don't care about the= ingestion time

    Thanks for the help!


    = --
    Mario Pastorelli | TERALYTICS

    software engineer

    Teralytics AG |=C2=A0Zolls= trasse 62 | 8005 Zurich=C2=A0| Switzerland=C2=A0
    phone:     = +41794381682
    email: mario.pastorelli@teralytics.ch
    www.teralytics.net

    Company registration number: CH-020.3.037.709-7 |= Trade register Canton Zurich
    Board of directors: Georg Polzer, Luc= iano Franceschina, Mark Schmitz, Yann de Vries

    This e-mail message contains confidential i= nformation which is for the sole attention and use of the intended recipient. Please notify us at once = if you think that it may not be intended for you and delete it immediately.





    --
    <= div>
    Mario Pastorelli | TERALYTICS

    software engineer

    Teralytics AG |=C2=A0Zollstrasse 62 | 8005 Zurich=C2=A0| Switzerla= nd=C2=A0
    phone:    <= font color=3D"#444444"> +41794381682=
    email: mario.pastorelli@teralytics.ch=
    www.teralytics.net    <= /p>

    Company registration n= umber: CH-020.3.037.709-7 | Trade register Canton Zurich
    Board of directors: Georg Polzer, Luc= iano Franceschina, Mark Schmitz, Yann de Vries

    This e-mail message contains confidential i= nformation which is for the sole attention and use of the intended recipient. Please notify us at once = if you think that it may not be intended for you and delete it immediately.


    --001a113dafd68d2e26053384b09b--