hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas Nair" <the...@hortonworks.com>
Subject Re: Review Request 15663: Hive should be able to skip header and footer rows when reading data file for a table
Date Fri, 22 Nov 2013 02:19:02 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15663/#review29253
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
<https://reviews.apache.org/r/15663/#comment56427>

    I think this code to separate header reading and initializing footer buffer a separate
function to improve readability. 



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
<https://reviews.apache.org/r/15663/#comment56424>

    can you make this a constant string and referce the constant variable name ?
    Unfortunately (for some historic reason that I don't understand) the place where all the
similar constants are being stored is in  serde/if/serde.thrift 



ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
<https://reviews.apache.org/r/15663/#comment56425>

    I think it is better fail here and let the user know, rather than use a smaller footer
skip lines than what user specified.
    



ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java
<https://reviews.apache.org/r/15663/#comment56426>

    how about using hive's ObjectPair instead ?



ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
<https://reviews.apache.org/r/15663/#comment56401>

    Sorry, I had failed to notice that this is a table property. But we need to document this
table property.
    



ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
<https://reviews.apache.org/r/15663/#comment56404>

    This does not handle the case where multiple splits are combined together.
    We need to disable combination of splits in CombineHiveInputFormat getSplits() call.
    
    



ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java
<https://reviews.apache.org/r/15663/#comment56422>

    Is this change for avoiding some NPE ?



ql/src/test/queries/clientpositive/file_with_header_footer.q
<https://reviews.apache.org/r/15663/#comment56423>

    can you add some negative tests where skip.footer.number exceeds hive.file.max.footer
?


- Thejas Nair


On Nov. 19, 2013, 1:31 a.m., Eric Hanson wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15663/
> -----------------------------------------------------------
> 
> (Updated Nov. 19, 2013, 1:31 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-5795
>     https://issues.apache.org/jira/browse/HIVE-5795
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hive should be able to skip header and footer rows when reading data file for a table
> 
> (I am uploading this on behalf of Shuaishuai Nie since he's not in the office)
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 32ab3d8 
>   data/files/header_footer_table_1/0001.txt PRE-CREATION 
>   data/files/header_footer_table_1/0002.txt PRE-CREATION 
>   data/files/header_footer_table_1/0003.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION 
>   data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION 
>   itests/qtest/pom.xml a453d8a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 5abcfc1 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java dd5cb6b

>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 0ec6e63 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java 85dd975

>   ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 0686d9b 
>   ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION 
>   ql/src/test/results/clientpositive/file_with_header_footer.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15663/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Eric Hanson
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message