From dev-return-41973-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Mon Nov 19 11:58:16 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E6515180671 for ; Mon, 19 Nov 2018 11:58:15 +0100 (CET) Received: (qmail 18140 invoked by uid 500); 19 Nov 2018 10:58:15 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 18128 invoked by uid 99); 19 Nov 2018 10:58:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Nov 2018 10:58:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E31441808BC for ; Mon, 19 Nov 2018 10:58:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.339 X-Spam-Level: X-Spam-Status: No, score=0.339 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-1.459, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 34T5FLHbZ3Z2 for ; Mon, 19 Nov 2018 10:58:12 +0000 (UTC) Received: from mail-vs1-f51.google.com (mail-vs1-f51.google.com [209.85.217.51]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 45051622F8 for ; Mon, 19 Nov 2018 10:58:12 +0000 (UTC) Received: by mail-vs1-f51.google.com with SMTP id y27so17520861vsi.1 for ; Mon, 19 Nov 2018 02:58:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=fB1yd870gVw3Kx0okJ7AUaBDo8s7vtKzZ4ysk850OSM=; b=IWR0XTXqC3i2onQvYen1R2fa/CYpPxtOjnq+e4A6kSSEaXzWV9ZzermGsDsXkDAFV9 NvFufOKBiC39gDesTT4Oo3Wfa2N9UoFmqIa7wydfXROPQ9XebL+gpRDovP53E2Rkbl2s 6/QYJ4JArAO5dUGsFxkW07eaWliV0vbl5/UnsoSCdU3Wm5D6Yh629EYRdSvs7r1G1PFk ARsQneTHorPkK4W3DljtB491wv+l6ssEDQcN9i5/uDnpeipHVcSz8A8EKkRQVNxGXv57 vSZpjTLLu1yAOpzZBaCk+FAo/fF0q4ZkO40r/4YJsEsAPEgeEjhvecbNNc8B4R1B7dY7 p9Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=fB1yd870gVw3Kx0okJ7AUaBDo8s7vtKzZ4ysk850OSM=; b=bwUb0Qsnv65gaWAyGhksO8Q7sELiEoBrpuauL5ydG4N0R4eBLzPm6QIFq0ITJbBlVN b1bpsUmxB2j0MQZWrcD82QvtTFvMX8mT+j6yBx08V+dgCy2ASKBNftv6FzeC9uV8BUgf TBccKXVgtcILdxZveC0Uk81r69bE6125OtiFoluzubU0NHrC5vVNm2Df+G1OLgw96asS WFjjyJ0zagZyIIPrGaU9lJgA1Nw6XUZvDYrBa/B6x2p+Lwt1ZgI6UYCkrADgjq1GPiol jtiSdwhFnNIM3wOThmleXRTCr23UJWE5SdOeTtU8jyfx1A4lPM5wcqo6I1cj5oNLdAfK PB8g== X-Gm-Message-State: AGRZ1gKdK90h768CrbTkhfmnJQNjwNhmD1GGihacLSFo5cT1nHKq5bpq FgRhVlU2QCpoFXHgglnW5Mbju7y7eZiY0f0txmWmmWsf X-Google-Smtp-Source: AJdET5fB0/wQgU7g8LxsOEMgHmBtjt8y18PZ2zlsCXcvdbeQ1aIbYETg9dKO/Erpikjj0a/RFQxzRRoe+HqFHl0WaYA= X-Received: by 2002:a67:7a44:: with SMTP id v65mr4988798vsc.190.1542625091574; Mon, 19 Nov 2018 02:58:11 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andrey Mashenkov Date: Mon, 19 Nov 2018 13:58:00 +0300 Message-ID: Subject: Re: Disk page compression for Ignite persistent store To: dev Content-Type: multipart/alternative; boundary="0000000000009ae0d6057b0264b1" --0000000000009ae0d6057b0264b1 Content-Type: text/plain; charset="UTF-8" Hi Sergi, It is not clear for me will your changes affect PageSnapshot WAL record. Is it possible to add compression support for PageSnapshot WAL record as well, to reduce WAL size? Thanks. On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin wrote: > Folks, > > I've implemented page compression for persistent store and going to merge > it to master. > > https://github.com/apache/ignite/pull/5200 > > Some design notes: > > It employs "hole punching" approach, it means that the pages are kept > uncompressed in memory, > but when they get written to disk, they will be compressed and all the > extra file system blocks for the page will be released. Thus the storage > files become sparse. > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which > basically just takes only meaningful data from half-filled pages but does > not apply any compression. It is easy to add more if needed. > > Since we can release only full file system blocks which are typically 4k > size, user must configure page size to be at least multiple FS blocks, e.g. > 8k or 16k. It also means that max compression ratio here is fsBlockSize / > pageSize = 4k / 16k = 0.25 > > It is possible to enable compression for existing databases if they were > configured for large enough page size. In this case pages will be written > to disk in compressed form when updated, and the database will become > compressed gradually. > > There will be 2 new properties on CacheConfiguration > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page > compression. > > Compression dictionaries are not supported at the time, but may in the > future. IMO it should be added as a separate feature if needed. > > The only supported platform for now is Linux. Since all popular file > systems support sparse files, it must be relatively easy to support more > platforms. > > Please take a look and provide your thoughts and suggestions. > > Thanks! > > Sergi > -- Best regards, Andrey V. Mashenkov --0000000000009ae0d6057b0264b1--