hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
Date Fri, 13 Jul 2018 07:00:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanth Jayachandran updated HIVE-20165:
-----------------------------------------
    Status: Patch Available  (was: Open)

> Enable ZLIB for streaming ingest
> --------------------------------
>
>                 Key: HIVE-20165
>                 URL: https://issues.apache.org/jira/browse/HIVE-20165
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 4.0.0, 3.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>         Attachments: HIVE-20165.1.patch
>
>
> Per [~gopalv]'s recommendation tried running streaming ingest with and without zlib.
Following are the numbers
>  
> *Compression: NONE*
> Total rows committed: 93800000
> Throughput: *1563333* rows/second
> [prasanth@cn105-10 culvert]$ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
> *14.1 G*  /apps/hive/warehouse/prasanth.db/culvert
>  
> *Compression: ZLIB*
> Total rows committed: 92100000
> Throughput: *1535000* rows/second
> [prasanth@cn105-10 culvert]$ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
> *7.4 G*  /apps/hive/warehouse/prasanth.db/culvert
>  
> ZLIB is getting us 2x compression and only 2% lesser throughput. We should enable ZLIB
by default for streaming ingest. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message