hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "churro morales (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16710) Add ZStandard Codec to Compression.java
Date Thu, 13 Oct 2016 20:04:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573013#comment-15573013
] 

churro morales commented on HBASE-16710:
----------------------------------------

[~saint.ack@gmail.com] hadoop will use the suffix zst to denote the file is compressed in
ZStandard format.  I have a patch up right now that does work with HBase but unfortunately
all of the block or chunked based compressors add a header in front of the chunk to denote
decompression size.  Thus compression libraries like snappy don't work when you compress outside
of the hadoop ecosystem.  I am currently working on a streaming based approach such that if
you compress through the cli and put the file into hdfs it will compress / decompress.  

I have tested the current patch by back porting to a hadoop 2.x branch with HBase and it works.
 The compression / decompression speed as well as compression ratios are very good.  It should
be a few weeks to finish up the streaming patch as I have other things on my plate. 

I'll update this JIRA once I get something upstream. 



> Add ZStandard Codec to Compression.java
> ---------------------------------------
>
>                 Key: HBASE-16710
>                 URL: https://issues.apache.org/jira/browse/HBASE-16710
>             Project: HBase
>          Issue Type: Task
>    Affects Versions: 2.0.0
>            Reporter: churro morales
>            Assignee: churro morales
>            Priority: Minor
>         Attachments: HBASE-16710-0.98.patch, HBASE-16710-1.2.patch, HBASE-16710.patch
>
>
> HADOOP-13578 is adding the ZStandardCodec to hadoop.  This is a placeholder to ensure
it gets added to hbase once this gets upstream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message