From dev-return-38162-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Aug 23 14:12:55 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DB60018061A for ; Thu, 23 Aug 2018 14:12:54 +0200 (CEST) Received: (qmail 87132 invoked by uid 500); 23 Aug 2018 12:12:53 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 87113 invoked by uid 99); 23 Aug 2018 12:12:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Aug 2018 12:12:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CA848180833 for ; Thu, 23 Aug 2018 12:12:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.87 X-Spam-Level: * X-Spam-Status: No, score=1.87 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id vwIeupfoQxCe for ; Thu, 23 Aug 2018 12:12:51 +0000 (UTC) Received: from mail-qk0-f195.google.com (mail-qk0-f195.google.com [209.85.220.195]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 805AC5F3CC for ; Thu, 23 Aug 2018 12:12:51 +0000 (UTC) Received: by mail-qk0-f195.google.com with SMTP id b19-v6so3402074qkc.6 for ; Thu, 23 Aug 2018 05:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=MSxjN5UIyW4j6BP4zLehYgOYKzcAbIGJTXlBYwjqq/o=; b=bkgD1tle+VEKDOD3ZLv+3nt7NKQEL+PFQUwvhABC1wy5uwIVapD6S82Akj8Jar4AAu g+Ex0QSpazsBUBmzJmxoyoex2JFUPxcNNABapKEe5LBIn6y6FxFKCoqv+HlUxNkHs6wd F2W203TNAuL+lcV68U0NyIzAQKcYWid25cvU05ySy55psUUg726JC/G+hAoCYrfAI/Qq 8dBxOQfV+N8YeGw52EEMiaAZ928O+81ww2zP3dzWYKabmz1zNQN40PX31wb5UncS6zjq 28zGBWSKIoh76yLUcBl6RITTZWAOI1qlwa4O4dXPTy12PW/wIRUIBBNkH5fGUG4LgsZz u8sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=MSxjN5UIyW4j6BP4zLehYgOYKzcAbIGJTXlBYwjqq/o=; b=uCKHbvUEC5/LGBrOoboAuBfN4cOXyVMXBAU61Jcrk/1k7H/kZqgeslWCt2auKUC30J f4qvJqXIZdHUn1OvQddqUl+Jzy0T1kbVrtUx/Mn/VCuM+UA21wkNqpkbYEZwe1WS3f1w bgTK7sVjKwPJpCpugppfGC48kpUIl5yZvHb2apxYv3efuQ41pVBzqyuyxDxhqB0RoO4h 27YRjYyFJrRdsEkR1EnckxIrDlLg2fOiJcLLwsnEispwmzyJ4Z+NWXIdWchE7W4NzH10 ubAmCFZsXKjUvEmj92kF55OnNeE+MRMAHqjhoey+9T/vbTFumy3UBw2N4LwrWlJ/Q36+ 7KWQ== X-Gm-Message-State: AOUpUlEH4DhmQHE2OHx4LBNNTqJ8t8rvIy84zZRlmyLoQySAVcavOyQ1 o34zHCpwzFpESBkr8748tl3Q2vGUS/chWHPe1GBr1A== X-Google-Smtp-Source: AA+uWPwJ3sEwNJmyTu5RBqy89QO6IGbnBH4GZ1TyUVgJS3JdrHFb+iB2K6xxga0Cbl45GY8UEqYRGAudLHR/2Efh5mY= X-Received: by 2002:a37:60c4:: with SMTP id u187-v6mr51900517qkb.143.1535026370886; Thu, 23 Aug 2018 05:12:50 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:aed:2663:0:0:0:0:0 with HTTP; Thu, 23 Aug 2018 05:12:50 -0700 (PDT) In-Reply-To: References: From: Ilya Kasnacheev Date: Thu, 23 Aug 2018 15:12:50 +0300 Message-ID: Subject: Re: Compression prototype To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="0000000000008ec0dd0574192d4f" --0000000000008ec0dd0574192d4f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello! Currently, the dictionary for decompression is only stored on heap. After restart there's compressed data in the PDS, but there's no dictionary :) Regards, --=20 Ilya Kasnacheev 2018-08-23 14:58 GMT+03:00 Dmitriy Pavlov : > Hi Ilya, > > Thank you for sharing this here. I believe this contribution will be > accepted by the Community. Moreover, it shows so remarkable performance > boost. > > I'm pretty sure this patch will be reviewed by Ignite Native Persistence > experts soon. > > What do you mean by can't survive PDS node restart? > > Sincerely, > Dmitriy Pavlov > > =D1=87=D1=82, 23 =D0=B0=D0=B2=D0=B3. 2018 =D0=B3. =D0=B2 13:09, Ilya Kasn= acheev : > > > Hello! > > > > I want to share with the developer community my compression prototype. > > > > Long story short, it compresses BinaryObject's byte[] as they are writt= en > > to Durable Memory page, operating on a pre-built dictionary. Typical > > compression ratio is 0.4 (meaning 2.5x compression) using custom > > LZW+Huffman. Metadata, indexes and primitive values are unaffected > > entirely. > > > > This is akin to DB2's table-level compression[1] but independently > > invented. > > > > On Yardstick tests performance hit is -6% with PDS and up to -25% (in > > throughput) with In-Memory loads. It also means you can fit ~twice as > much > > data into the same IM cluster, or have higher ram/disk ratio with PDS > > cluster, saving on hardware or decreasing latency. > > > > The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=3Dtr= ue > to > > activate). Note that it will not presently survive a PDS node restart. > > The impact is very small, the patch should be applicable to most 2.x > > releases. > > > > Sure there's a long way before this prototype can have hope of being > > included, but first I would like to hear input from fellow igniters. > > > > See also IEP-20[3]. > > > > 1. > > > > https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10. > 5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html > > 2. https://github.com/apache/ignite/pull/4295 > > 3. > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 20%3A+Data+Compression+in+Ignite > > > > Regards, > > > > -- > > Ilya Kasnacheev > > > --0000000000008ec0dd0574192d4f--