hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2370) Improve RCFileCat performance significantly
Date Thu, 11 Aug 2011 22:46:29 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083768#comment-13083768
] 

jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/
-----------------------------------------------------------

(Updated 2011-08-11 22:44:48.620762)


Review request for hive, Yongqiang He, Ning Zhang, and namit jain.


Summary (updated)
-------

This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to
5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this
performance:

Initial:
0.32 MB/s

Change System.out to use bigger buffer (not line buffered)
1.7MB/s

Unchecked Get:
1.75MB/s

Use StringBuilder to construct each row before writing output.
3.7MB/s

Streamline decoding:
4.16 MB/s

Use StringBuilder to buffer multiple lines:
5 MB/s

Tuning buffer sizes:
5.15 MB/s


I also added a --verbose mode which writes progress updates to stderr.


This addresses bug HIVE-2370.
    https://issues.apache.org/jira/browse/HIVE-2370


Diffs
-----

  trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839 

Diff: https://reviews.apache.org/r/1474/diff


Testing
-------

Used diff to check output was same as old version of RCFileCat


Thanks,

Tim



> Improve RCFileCat performance significantly
> -------------------------------------------
>
>                 Key: HIVE-2370
>                 URL: https://issues.apache.org/jira/browse/HIVE-2370
>             Project: Hive
>          Issue Type: Improvement
>          Components: CLI
>    Affects Versions: 0.8.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Minor
>         Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of
compressed RCFile.  We can implement much faster version to enable faster export of data from
Hive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message