From dev-return-38196-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Aug 24 01:56:43 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 718D918061A for ; Fri, 24 Aug 2018 01:56:43 +0200 (CEST) Received: (qmail 35084 invoked by uid 500); 23 Aug 2018 23:56:42 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 35071 invoked by uid 99); 23 Aug 2018 23:56:42 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Aug 2018 23:56:42 +0000 Received: from mail-qt0-f170.google.com (mail-qt0-f170.google.com [209.85.216.170]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id A36AA4EEF for ; Thu, 23 Aug 2018 23:56:41 +0000 (UTC) Received: by mail-qt0-f170.google.com with SMTP id x7-v6so8353579qtk.5 for ; Thu, 23 Aug 2018 16:56:41 -0700 (PDT) X-Gm-Message-State: AOUpUlHV/FBEmn92Zo6/PQMYStHf1OwKhrbjUIj/qtbAjUalzW7vR5mr VhIW1Ogy9mvDVg+fmrVdT6ywSQL2uT+En5tAvEnM0A== X-Google-Smtp-Source: AA+uWPxo2pO3Bfoic3pHwIVBeDqq4VTvkZumSzk5bpbJWJ5qdk3sBeZQlQez0NVo51W9XTQbVzgLHnTzJg8h80uRAXg= X-Received: by 2002:ac8:528c:: with SMTP id s12-v6mr22717467qtn.311.1535068601071; Thu, 23 Aug 2018 16:56:41 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Denis Magda Date: Thu, 23 Aug 2018 16:56:13 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Compression prototype To: dev Content-Type: multipart/alternative; boundary="000000000000ac43f805742302eb" --000000000000ac43f805742302eb Content-Type: text/plain; charset="UTF-8" Hi Ilya, Sounds terrific! Is this part of the following Ignite enhancement proposal? https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite -- Denis On Thu, Aug 23, 2018 at 5:17 AM Ilya Kasnacheev wrote: > Hello! > > My plan was to add a compression section to cache configuration, where you > can enable compression, enable key compression (which has heavier > performance implications), adjust dictionary gathering settings, and in the > future possibly choose betwen algorithms. In fact I'm not sure, since my > assumption is that you can always just use latest&greatest, but maybe we > can have e.g. very fast and not very strong vs. slower but stronger one. > > I'm not sure yet if we should share dictionary between all caches vs. > having separate dictionary for every cache. > > With regards to data format, of course there will be room for further > extension. > > Regards, > > -- > Ilya Kasnacheev > > 2018-08-23 15:13 GMT+03:00 Sergey Kozlov : > > > Hi Ilya > > > > Is there a plan to introduce it as an option of Ignite configuration? In > > that instead the boolean type I suggest to use the enum and reserve the > > ability to extend compressions algorithms in future > > > > On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev < > > ilya.kasnacheev@gmail.com> > > wrote: > > > > > Hello! > > > > > > I want to share with the developer community my compression prototype. > > > > > > Long story short, it compresses BinaryObject's byte[] as they are > written > > > to Durable Memory page, operating on a pre-built dictionary. Typical > > > compression ratio is 0.4 (meaning 2.5x compression) using custom > > > LZW+Huffman. Metadata, indexes and primitive values are unaffected > > > entirely. > > > > > > This is akin to DB2's table-level compression[1] but independently > > > invented. > > > > > > On Yardstick tests performance hit is -6% with PDS and up to -25% (in > > > throughput) with In-Memory loads. It also means you can fit ~twice as > > much > > > data into the same IM cluster, or have higher ram/disk ratio with PDS > > > cluster, saving on hardware or decreasing latency. > > > > > > The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=true > > to > > > activate). Note that it will not presently survive a PDS node restart. > > > The impact is very small, the patch should be applicable to most 2.x > > > releases. > > > > > > Sure there's a long way before this prototype can have hope of being > > > included, but first I would like to hear input from fellow igniters. > > > > > > See also IEP-20[3]. > > > > > > 1. > > > https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10. > > > 5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html > > > 2. https://github.com/apache/ignite/pull/4295 > > > 3. > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > 20%3A+Data+Compression+in+Ignite > > > > > > Regards, > > > > > > -- > > > Ilya Kasnacheev > > > > > > > > > > > -- > > Sergey Kozlov > > GridGain Systems > > www.gridgain.com > > > --000000000000ac43f805742302eb--