Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============1097961620067925402=="
MIME-Version: 1.0
Subject: Re: Review Request 17899: HIVE-5998 Add vectorized reader for Parquet
 files
From: "Brock Noland" <brock@cloudera.com>
To: "Jitendra Pandey" <jitendra@hortonworks.com>,
 "Eric Hanson" <ehans@microsoft.com>, "Brock Noland" <brock@cloudera.com>
Cc: "Remus Rusanu" <remusr@microsoft.com>, "hive" <dev@hive.apache.org>
Date: Thu, 13 Feb 2014 17:14:40 -0000
Message-ID: <20140213171440.22558.22918@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Brock Noland" <noreply@reviews.apache.org>
References: <20140213083808.22558.88197@reviews.apache.org>
In-Reply-To: <20140213083808.22558.88197@reviews.apache.org>
Reply-To: "Brock Noland" <brock@cloudera.com>

--===============1097961620067925402==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17899/#review34382
-----------------------------------------------------------


Nice work!! I noted parts of the patch have 4 spaces and other parts have two. 


ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
<https://reviews.apache.org/r/17899/#comment64462>

    trailing ws


ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
<https://reviews.apache.org/r/17899/#comment64463>

    trailing ws


- Brock Noland


On Feb. 13, 2014, 8:38 a.m., Remus Rusanu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17899/
> -----------------------------------------------------------
> 
> (Updated Feb. 13, 2014, 8:38 a.m.)
> 
> 
> Review request for hive, Brock Noland, Eric Hanson, and Jitendra Pandey.
> 
> 
> Bugs: HIVE-5998
>     https://issues.apache.org/jira/browse/HIVE-5998
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Implementation is straight forward and very simple, but offers all benefits of vectorization possible with a 'shallow' vectorized reader (ie. one that doe not got into parquet-mr project changes). the only complication arrised because of discrepancies between the object inspector seen by the inputformat and the actual output provided by the Parquet readers (eg. OI declares 'byte' primitives but the Parquet reader outputs IntWritable). I had to create a just-in-time VectorColumnAssigner colelciton base don whatever writers the Parquet record reader provides. It is assumed the reader does not change it's output during the iteration.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java d1a75df 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java 0b504de 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java d409d44 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java d3412df 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java PRE-CREATION 
>   ql/src/test/queries/clientpositive/vectorized_parquet.q PRE-CREATION 
>   ql/src/test/results/clientpositive/vectorized_parquet.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/17899/diff/
> 
> 
> Testing
> -------
> 
> Manually tested. New query .q added.
> 
> 
> Thanks,
> 
> Remus Rusanu
> 
>


--===============1097961620067925402==--