From dev-return-38651-archive-asf-public=cust-asf.ponee.io@ignite.apache.org  Tue Sep  4 16:16:19 2018
Return-Path: <dev-return-38651-archive-asf-public=cust-asf.ponee.io@ignite.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by mx-eu-01.ponee.io (Postfix) with SMTP id 82DA0180629
	for <archive-asf-public@cust-asf.ponee.io>; Tue,  4 Sep 2018 16:16:18 +0200 (CEST)
Received: (qmail 7552 invoked by uid 500); 4 Sep 2018 14:16:17 -0000
Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@ignite.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@ignite.apache.org>
List-Post: <mailto:dev@ignite.apache.org>
List-Id: <dev.ignite.apache.org>
Reply-To: dev@ignite.apache.org
Delivered-To: mailing list dev@ignite.apache.org
Received: (qmail 7541 invoked by uid 99); 4 Sep 2018 14:16:17 -0000
Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Sep 2018 14:16:17 +0000
Received: from mail-oi0-f41.google.com (mail-oi0-f41.google.com [209.85.218.41])
	by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 81654D26
	for <dev@ignite.apache.org>; Tue,  4 Sep 2018 14:16:16 +0000 (UTC)
Received: by mail-oi0-f41.google.com with SMTP id k12-v6so6958952oiw.8
        for <dev@ignite.apache.org>; Tue, 04 Sep 2018 07:16:16 -0700 (PDT)
X-Gm-Message-State: APzg51AZlefMX8Fw7vdheLlf4/W2fBtB7hQVC7vfW8MZTx7vklWifNSM
	0RJcC1WbysncvRK5TpeC8W7oelflP+xs/SBqSa/sCw==
X-Google-Smtp-Source: ANB0VdaAnGwpag0WnXi4b8LSTK32S/jZDHvNtiN7jSykoRgF3azViu5nH9WQzxXJoBWh0batPYj+UGAtt2IfzhC9bGc=
X-Received: by 2002:aca:5a45:: with SMTP id o66-v6mr10352222oib.155.1536070575739;
 Tue, 04 Sep 2018 07:16:15 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:ac9:2927:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 07:15:34 -0700 (PDT)
In-Reply-To: <CAA-MoLhZpBy+b7fL8xdoWJTsvbYCRPg8N_Gy5EfWQMX8OR7ZAA@mail.gmail.com>
References: <CAMbfdjbZYJpCLFOKBYXd=+=zR59y-oPaNeJaCgtWfg45waOR7w@mail.gmail.com>
 <CAA-MoLhLBTuo8Xh6BxWBTk7cag-_fCCoXqxxFP-1RH_W37Dbqw@mail.gmail.com>
 <CAK0qHnrvyCT5cJJy7hKTjXXRBeA5Qc=bL3e0B5sVeDmURd=xzg@mail.gmail.com>
 <CAA-MoLgPi5JvVLrgCe28O_oCQg28arvmkg0=zooZh9ztzjUbQw@mail.gmail.com>
 <CAK0qHnp0uajg9ogPuZUbSHpeu6c6FaPhN2+KV3t_id6kVKHtiQ@mail.gmail.com>
 <CAFyTW-irJ3fLre272kpbDJ3Xe3UkugKrFguteJb+YSjSadKMnA@mail.gmail.com>
 <CAA-MoLg7MoS36GFgc_zVp-f0NjvBqRUNHw=JYOWRw895Vu7HMQ@mail.gmail.com>
 <CAFyTW-hZjQOLfp4rA4v4W7_F38M7z-R0GJH-ho1Fzp=D4s6JzQ@mail.gmail.com>
 <CAA-MoLhDKrPodk12Tu5obWXtW=LZBikrX4gwg-7VyMDCPyruZw@mail.gmail.com>
 <CAA-MoLiHJ0VEwpEBTQ7nhVvG64xo2Ke6vwaLB2=k+Hzbhrvrcg@mail.gmail.com>
 <1535996189536-0.post@n4.nabble.com> <CA+0=VoUt2io=qFja29aN=_4CHZ30aLvZCBQh1Hw0StNWL0p3rg@mail.gmail.com>
 <CAA-MoLifu2Xm9p=2RoA_9T3_PDKRGKZ8fmf5VY1bfwrVFOq_3A@mail.gmail.com>
 <CA+0=VoW2mY=CDg8gquQ+5yOdQZR6obKEGBv-hFOHjjm9obfi7A@mail.gmail.com> <CAA-MoLhZpBy+b7fL8xdoWJTsvbYCRPg8N_Gy5EfWQMX8OR7ZAA@mail.gmail.com>
From: Dmitriy Setrakyan <dsetrakyan@apache.org>
Date: Tue, 4 Sep 2018 07:15:34 -0700
X-Gmail-Original-Message-ID: <CA+0=VoVtGegUaKiq5G9yxqF8XEoyLyiw1e0XdYc5NyQga4LV8A@mail.gmail.com>
Message-ID: <CA+0=VoVtGegUaKiq5G9yxqF8XEoyLyiw1e0XdYc5NyQga4LV8A@mail.gmail.com>
Subject: Re: Compression prototype
To: dev <dev@ignite.apache.org>
Content-Type: multipart/alternative; boundary="000000000000047b3605750c4da7"

--000000000000047b3605750c4da7
Content-Type: text/plain; charset="UTF-8"

On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
wrote:

> Hello!
>
> Each node has a local dictionary (per node currently, per cache planned).
> Dictionary is never shared between nodes. As data patterns shift,
> dictionary rotation is also planned.
>
> With Zstd, the best dictionary size seems to be 1024 bytes. I imagine It is
> enough to store common BinaryObject boilerplate, and everything else is
> compressed on the fly. The source sample is 16k records.
>
>
Thanks, Ilya, understood. I think per-cache is a better idea. However, I
have a question about dictionary size. Ignite stores TBs of data. How do
you plan the dictionary to fit in 1K bytes?

D.

--000000000000047b3605750c4da7--