hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13578) Add Codec for ZStandard Compression
Date Wed, 14 Dec 2016 21:35:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749557#comment-15749557

Jason Lowe commented on HADOOP-13578:

Thanks for updating the patch!  It's really close now.

We need to check for an error if ZSTD_endStream returns a failure, otherwise we could end
up looping forever on a persistent error being returned from ZSTD_endStream.  The outer compress
logic will keep calling compress with finish=true trying to wrap things up and the JNI code
will keep avoiding setting the finished flag because ZSTD_endStream isn't returning 0.

There's no return after the THROW macros here, so we can fall through and call ZSTD_flushStream
after ZSTD_compressStream returns an error.  That means the error being reported from compress
might be clobbered by the error being returned from flush if they are different.  Also we
can just move the error check on "size" up inside the flush block since it only applies to
that case, or change "remaining_to_flush" to "size" and let the previous point be fixed by
falling through to the common error check on "size".
    size_t size = dlsym_ZSTD_compressStream(stream, &output, &input);
    if (dlsym_ZSTD_isError(size)) {
        THROW(env, "java/lang/InternalError", dlsym_ZSTD_getErrorName(size));
    if (finish && input.pos == input.size) {
        // end the stream, flush and  write the frame epilogue
        size_t const remaining_to_flush = dlsym_ZSTD_endStream(stream, &output);
        if (!remaining_to_flush) {
            (*env)->SetBooleanField(env, this, ZStandardCompressor_finished, JNI_TRUE);
    else {
        // need to flush the output buffer
        // this also updates the output buffer position.
        size = dlsym_ZSTD_flushStream(stream, &output);
    if (dlsym_ZSTD_isError(size)) {
        THROW(env, "java/lang/InternalError", dlsym_ZSTD_getErrorName(size));

> Add Codec for ZStandard Compression
> -----------------------------------
>                 Key: HADOOP-13578
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13578
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: churro morales
>            Assignee: churro morales
>         Attachments: HADOOP-13578.patch, HADOOP-13578.v1.patch, HADOOP-13578.v2.patch,
HADOOP-13578.v3.patch, HADOOP-13578.v4.patch, HADOOP-13578.v5.patch, HADOOP-13578.v6.patch,
HADOOP-13578.v7.patch, HADOOP-13578.v8.patch
> ZStandard: https://github.com/facebook/zstd has been used in production for 6 months
by facebook now.  v1.0 was recently released.  Create a codec for this library.  

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message