drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinfeng Ni" <...@maprtech.com>
Subject Re: Review Request 38796: DRILL-3209: Support reading Hive tables using Drill's native parquet reader
Date Tue, 29 Sep 2015 19:59:02 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38796/#review100998
-----------------------------------------------------------

Ship it!


The revised comments address my comments.


contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
(line 267)
<https://reviews.apache.org/r/38796/#comment158300>

    I have one question about partition column. 
    
    Let's say Hive has 'year" as partition column. For value 2015, does Hive put "year=2015"
as the directory name? If that's the case, then "year=2015" would be treated as "dir0" by
native parquet reader, in stead of "2015"? Do we need handle the difference of partition column
between hive scan and native scan?


- Jinfeng Ni


On Sept. 29, 2015, 9:23 a.m., Venki Korukanti wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38796/
> -----------------------------------------------------------
> 
> (Updated Sept. 29, 2015, 9:23 a.m.)
> 
> 
> Review request for drill and Jinfeng Ni.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> Please jira DRILL-3209 for details.
> 
> 
> Diffs
> -----
> 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HivePartitionDescriptor.java
11c6455 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
PRE-CREATION 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveDrillNativeParquetScan.java
PRE-CREATION 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveDrillNativeParquetSubScan.java
PRE-CREATION 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveDrillNativeScanBatchCreator.java
PRE-CREATION 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java
9ada569 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java
23aa37f 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveSubScan.java
2181c2a 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/DrillHiveTable.java
b459ee4 
>   contrib/storage-hive/core/src/test/java/org/apache/drill/exec/TestHivePartitionPruning.java
f0b4bdc 
>   contrib/storage-hive/core/src/test/java/org/apache/drill/exec/TestHiveProjectPushDown.java
6423a36 
>   contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestHiveStorage.java
9211af6 
>   contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestInfoSchemaOnHiveStorage.java
6118be5 
>   contrib/storage-hive/core/src/test/java/org/apache/drill/exec/store/hive/HiveTestDataGenerator.java
34a7ed6 
>   exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java 66f9f03 
>   exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
5838bd1 
> 
> Diff: https://reviews.apache.org/r/38796/diff/
> 
> 
> Testing
> -------
> 
> Added unittests to test reading all supported types, project pushdown and partition pruning.
Manually tested with Hive tables containing large amount of data (these tests will become
part of the regression suite).
> 
> 
> Thanks,
> 
> Venki Korukanti
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message