Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 14 Dec 2016 21:35:58 +0000 (UTC)
From: "Jason Lowe (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13002509.1472858578000.523660.1481751358438@Atlassian.JIRA>
In-Reply-To: <JIRA.13002509.1472858578000@Atlassian.JIRA>
References: <JIRA.13002509.1472858578000@Atlassian.JIRA> <JIRA.13002509.1472858578627@arcas>
Subject: [jira] [Commented] (HADOOP-13578) Add Codec for ZStandard
 Compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 14 Dec 2016 21:36:00 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749557#comment-15749557 ] 

Jason Lowe commented on HADOOP-13578:
-------------------------------------

Thanks for updating the patch!  It's really close now.

We need to check for an error if ZSTD_endStream returns a failure, otherwise we could end up looping forever on a persistent error being returned from ZSTD_endStream.  The outer compress logic will keep calling compress with finish=true trying to wrap things up and the JNI code will keep avoiding setting the finished flag because ZSTD_endStream isn't returning 0.

There's no return after the THROW macros here, so we can fall through and call ZSTD_flushStream after ZSTD_compressStream returns an error.  That means the error being reported from compress might be clobbered by the error being returned from flush if they are different.  Also we can just move the error check on "size" up inside the flush block since it only applies to that case, or change "remaining_to_flush" to "size" and let the previous point be fixed by falling through to the common error check on "size".
{code}
    size_t size = dlsym_ZSTD_compressStream(stream, &output, &input);
    if (dlsym_ZSTD_isError(size)) {
        THROW(env, "java/lang/InternalError", dlsym_ZSTD_getErrorName(size));
    }
    if (finish && input.pos == input.size) {
        // end the stream, flush and  write the frame epilogue
        size_t const remaining_to_flush = dlsym_ZSTD_endStream(stream, &output);
        if (!remaining_to_flush) {
            (*env)->SetBooleanField(env, this, ZStandardCompressor_finished, JNI_TRUE);
        }
    }
    else {
        // need to flush the output buffer
        // this also updates the output buffer position.
        size = dlsym_ZSTD_flushStream(stream, &output);
    }
    if (dlsym_ZSTD_isError(size)) {
        THROW(env, "java/lang/InternalError", dlsym_ZSTD_getErrorName(size));
    }
{code}

> Add Codec for ZStandard Compression
> -----------------------------------
>
>                 Key: HADOOP-13578
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13578
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: churro morales
>            Assignee: churro morales
>         Attachments: HADOOP-13578.patch, HADOOP-13578.v1.patch, HADOOP-13578.v2.patch, HADOOP-13578.v3.patch, HADOOP-13578.v4.patch, HADOOP-13578.v5.patch, HADOOP-13578.v6.patch, HADOOP-13578.v7.patch, HADOOP-13578.v8.patch
>
>
> ZStandard: https://github.com/facebook/zstd has been used in production for 6 months by facebook now.  v1.0 was recently released.  Create a codec for this library.  


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org