orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: [Discussion] Base 128 variable integer encoding is not always good
Date Tue, 18 Sep 2018 22:37:59 GMT
Hi,

>  From above observation, we find that it is better to disable LEB128 encoding while zstd
is used.

You can enable file size optimizations (automatically recommend better layouts for compression)
when 

"orc.encoding.strategy"="COMPRESSION"

There are a bunch of bitpacking loops that's controlled by that flag already.

>     https://github.com/facebook/zstd/issues/1325.

If I understand that correctly, a DIRECT_V2 would also work fine for the numeric sequences
in Zstd instead?

Cheers,
Gopal
 



Mime
View raw message