hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergio Pena" <sergio.p...@cloudera.com>
Subject Re: Review Request 35950: HIVE-11131: Get row information on DataWritableWriter once for better writing performance
Date Tue, 07 Jul 2015 16:25:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35950/
-----------------------------------------------------------

(Updated July 7, 2015, 4:25 p.m.)


Review request for hive, Ryan Blue, cheng xu, and Dong Chen.


Changes
-------

Address feedback changes.


Bugs: HIVE-11131
    https://issues.apache.org/jira/browse/HIVE-11131


Repository: hive-git


Description
-------

Implemented data type writers that will be created before the first Hive row is written to
Parquet. These writers contain information about object inspectors and schema of a specific
data type, and calls the specific addXXXX() method used by Parquet for each data type.


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java c195c3ec3ddae19bf255fc2c9633f8bf4390f428


Diff: https://reviews.apache.org/r/35950/diff/


Testing
-------

Tests from TestDataWritableWriter run OK.

I run other tests with micro-becnhmarks, and I got some better results from this new implemntation:

Using repeated rows across the file, this is the throughput increase using 1 million records:

bigint	boolean	double	float	int	    string
7.598	7.491	7.488	7.588	7.53	0.270     (before)
10.137	11.511	10.155	10.297	10.242  0.286     (after)

Using random rows across the file, the is the throughput increase using 1 million records:

bigint	boolean	double	float	int	    string
5.268	7.723	4.107	4.173	4.729	0.20       (before)
6.236	10.466	5.944	4.749	5.234	0.22       (after)


Thanks,

Sergio Pena


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message