ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergi Vladykin <sergi.vlady...@gmail.com>
Subject Re: Disk page compression for Ignite persistent store
Date Mon, 19 Nov 2018 14:00:15 GMT
Ilya,

Zstd itself has default compression level 3. I just used that number to be
consistent with the library defaults.

I will check if there is a significant difference in performance.

Sergi

пн, 19 нояб. 2018 г. в 14:59, Ilya Kasnacheev <ilya.kasnacheev@gmail.com>:

> Hello!
>
> You have zstd default level of 3. In my tests, zstd usually performed much
> better with compression level 2. Please consider.
>
> I admire your effort!
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 19 нояб. 2018 г. в 14:02, Sergi Vladykin <sergi.vladykin@gmail.com>:
>
> > Right now the functionality has nothing to do with WAL, but your idea
> > definitely makes sense and worth being implemented as a next step.
> >
> > Sergi
> >
> > пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov <
> andrey.mashenkov@gmail.com
> > >:
> >
> > > Hi Sergi,
> > >
> > > It is not clear for me will your changes affect PageSnapshot WAL
> record.
> > > Is it possible to add compression support for PageSnapshot WAL record
> as
> > > well, to reduce WAL size?
> > >
> > > Thanks.
> > >
> > > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin <
> sergi.vladykin@gmail.com
> > >
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > I've implemented page compression for persistent store and going to
> > merge
> > > > it to master.
> > > >
> > > > https://github.com/apache/ignite/pull/5200
> > > >
> > > > Some design notes:
> > > >
> > > > It employs "hole punching" approach, it means that the pages are kept
> > > > uncompressed in memory,
> > > > but when they get written to disk, they will be compressed and all
> the
> > > > extra file system blocks for the page will be released. Thus the
> > storage
> > > > files become sparse.
> > > >
> > > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY
> and
> > > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE,
> > which
> > > > basically just takes only meaningful data from half-filled pages but
> > does
> > > > not apply any compression. It is easy to add more if needed.
> > > >
> > > > Since we can release only full file system blocks which are typically
> > 4k
> > > > size, user must configure page size to be at least multiple FS
> blocks,
> > > e.g.
> > > > 8k or 16k. It also means that max compression ratio here is
> > fsBlockSize /
> > > > pageSize = 4k / 16k = 0.25
> > > >
> > > > It is possible to enable compression for existing databases if they
> > were
> > > > configured for large enough page size. In this case pages will be
> > written
> > > > to disk in compressed form when updated, and the database will become
> > > > compressed gradually.
> > > >
> > > > There will be 2 new properties on CacheConfiguration
> > > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup
> disk
> > > page
> > > > compression.
> > > >
> > > > Compression dictionaries are not supported at the time, but may in
> the
> > > > future. IMO it should be added as a separate feature if needed.
> > > >
> > > > The only supported platform for now is Linux. Since all popular file
> > > > systems support sparse files, it must be  relatively easy to support
> > more
> > > > platforms.
> > > >
> > > > Please take a look and provide your thoughts and suggestions.
> > > >
> > > > Thanks!
> > > >
> > > > Sergi
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrey V. Mashenkov
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message