apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2487) Malhar should support outputting data in Snappy compression
Date Fri, 05 May 2017 21:50:04 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999009#comment-15999009

ASF GitHub Bot commented on APEXMALHAR-2487:

GitHub user ilganeli opened a pull request:


    APEXMALHAR-2487 Added support for Snappy compression in FilterStreamProvider

    * Based on existing code to output Gzip or CipherText this patch adds support for writing
data out as Hadoop-readable Snappy format
    * Added unit tests which validate both the provider and the simpler SnappyStream functionality.
    * This patch reuses some code from existing tests where possible.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ilganeli/incubator-apex-malhar APEXMALHAR-2487

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #616
commit e756309c676778917ae0464c1ec44a9cf84fbe2b
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Date:   2017-04-29T04:54:06Z

    Added support for Snappy compression in FilterStreamProvider, which in turn enables Snappy

commit 9ed1e7e3e0bb357f4cdad0b3cfc34cc40404d9f3
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Date:   2017-04-29T04:58:55Z

    Added additional check for presence of native Snappy libraries.

commit 9e442c8f41284d13fef1f2ede99a17f7e105c06d
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Date:   2017-04-29T05:18:46Z

    Checkstyle fixes.

commit 4f19d45d98f88d17dcde2e3b040b68f9f8a7f1b8
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Date:   2017-04-29T05:41:10Z

    Fixed header.

commit 70a30e6ef855afb5cb5399a84c271fa93ef23c05
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Date:   2017-05-05T21:48:14Z

    Adressed PR comments.


> Malhar should support outputting data in Snappy compression
> -----------------------------------------------------------
>                 Key: APEXMALHAR-2487
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2487
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: Ilya Ganelin
>            Assignee: Ilya Ganelin
> At present, the default file output operator (AbstractFileOutputOperator) supports compression
by setting the FilterStreamProvider. However, Malhar presently only includes two FilterStreamProvider
- one to Cipher data, and one for Gzip. 
> Snappy offers substantially improved performance over Gzip in terms of compression and
decompression speed at the expense of compression ratio. In certain applications this is useful.
Thus, it would be helpful to add a Snappy FilterStreamProvider.

This message was sent by Atlassian JIRA

View raw message