orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiening Dai <xndai....@live.com>
Subject Re: Zstd decoder support
Date Sat, 05 May 2018 18:46:40 GMT
>>> BTW we are about to do a release that fixes a bug with zstd.

I am curious which bug you are referring to. Is it a bug with Java implementation or it affects
C++ as well?


> On May 4, 2018, at 7:56 PM, Dain Sundstrom <dain@iq80.com> wrote:
> 
> 0.11 is released.
> 
> -dain
> 
> Sent from my iPhone
> 
>> On May 4, 2018, at 1:41 PM, Owen O'Malley <owen.omalley@gmail.com> wrote:
>> 
>> I just upgraded ORC to use aircompressor 0.10. I assume we'll want to move
>> to 0.11 before we use zstd?
>> 
>> .. Owen
>> 
>>> On Fri, May 4, 2018 at 12:49 PM, Dain Sundstrom <dain@iq80.com> wrote:
>>> 
>>> The maintained location (and the version we use in prod) is:
>>> https://github.com/airlift/aircompressor/tree/master/src/
>>> main/java/io/airlift/compress/zstd
>>> 
>>> We plan on writing the compressor soon as we need it for our production
>>> systems.
>>> 
>>> BTW we are about to do a release that fixes a bug with zstd.
>>> 
>>> -dain
>>> 
>>>> On May 4, 2018, at 11:19 AM, Xiening Dai <xndai.git@live.com> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> I think the major reason that we don’t support zstd compressor today is
>>> that there’s no native java library currently. But I do see a java
>>> decompressor in presto code base -
>>>> 
>>>> https://github.com/prestodb/presto/blob/8f4e5bb9340890f01291ee1b777a1b
>>> 2b921a90c4/presto-orc/src/main/java/com/facebook/presto/
>>> orc/zstd/ZstdDecompressor.java
>>>> 
>>>> I wonder if we can just integrate this so at least we have the
>>> decompress ability. I believe this is valuable as there are Orc files
>>> compressed in zstd, including some presto workload, and some of the
>>> workload in our installation (generated by c++ writer with zstd
>>> compression).
>>>> 
>>>> 
>>> 
>>> 

Mime
View raw message