ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Disk page compression for Ignite persistent store
Date Mon, 19 Nov 2018 11:58:46 GMT
Hello!

You have zstd default level of 3. In my tests, zstd usually performed much
better with compression level 2. Please consider.

I admire your effort!

Regards,
-- 
Ilya Kasnacheev


пн, 19 нояб. 2018 г. в 14:02, Sergi Vladykin <sergi.vladykin@gmail.com>:

> Right now the functionality has nothing to do with WAL, but your idea
> definitely makes sense and worth being implemented as a next step.
>
> Sergi
>
> пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov <andrey.mashenkov@gmail.com
> >:
>
> > Hi Sergi,
> >
> > It is not clear for me will your changes affect PageSnapshot WAL record.
> > Is it possible to add compression support for PageSnapshot WAL record as
> > well, to reduce WAL size?
> >
> > Thanks.
> >
> > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin <sergi.vladykin@gmail.com
> >
> > wrote:
> >
> > > Folks,
> > >
> > > I've implemented page compression for persistent store and going to
> merge
> > > it to master.
> > >
> > > https://github.com/apache/ignite/pull/5200
> > >
> > > Some design notes:
> > >
> > > It employs "hole punching" approach, it means that the pages are kept
> > > uncompressed in memory,
> > > but when they get written to disk, they will be compressed and all the
> > > extra file system blocks for the page will be released. Thus the
> storage
> > > files become sparse.
> > >
> > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE,
> which
> > > basically just takes only meaningful data from half-filled pages but
> does
> > > not apply any compression. It is easy to add more if needed.
> > >
> > > Since we can release only full file system blocks which are typically
> 4k
> > > size, user must configure page size to be at least multiple FS blocks,
> > e.g.
> > > 8k or 16k. It also means that max compression ratio here is
> fsBlockSize /
> > > pageSize = 4k / 16k = 0.25
> > >
> > > It is possible to enable compression for existing databases if they
> were
> > > configured for large enough page size. In this case pages will be
> written
> > > to disk in compressed form when updated, and the database will become
> > > compressed gradually.
> > >
> > > There will be 2 new properties on CacheConfiguration
> > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk
> > page
> > > compression.
> > >
> > > Compression dictionaries are not supported at the time, but may in the
> > > future. IMO it should be added as a separate feature if needed.
> > >
> > > The only supported platform for now is Linux. Since all popular file
> > > systems support sparse files, it must be  relatively easy to support
> more
> > > platforms.
> > >
> > > Please take a look and provide your thoughts and suggestions.
> > >
> > > Thanks!
> > >
> > > Sergi
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message