drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parth Chandra" <pchan...@maprtech.com>
Subject Review Request 28476: DRILL-1738: Allow case insensitive read in complex parquet reader
Date Wed, 26 Nov 2014 20:19:45 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28476/
-----------------------------------------------------------

Review request for drill, Jason Altekruse and Steven Phillips.


Repository: drill-git


Description
-------

The complex parquet reader looks up the type of a field by querying the parquet library which
does a case sensitive match on names. If a field is spelled using diffent case, the field
is not found and the parquet reader handles it as a missing field. If it was the only field,
the reader returns all the fields. 
The fix is to do a case insensitive name match. The fix also requires retaining both the type
name in the parquet file and the field name used by the user and propagating them down to
the creation of value vectors.


Diffs
-----

  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetGroupConverter.java
c6310b1 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java
c3e8330 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetRecordMaterializer.java
f9c3480 

Diff: https://reviews.apache.org/r/28476/diff/


Testing
-------

Tested the following queries on a complex file

select * from dfs.`complex.parquet` d;
select d.ingredients from dfs.`complex.parquet` d;
select d.ingredients.bag from dfs.`complex.parquet` d;
select d.ingredients.bag[0].name from dfs.`complex.parquet` d;

select d.ingredienTs from dfs.`complex.parquet` d;

select d.ingredients.Bag from dfs.`complex.parquet` d;
select d.Ingredients.bag from dfs.`complex.parquet` d;
select d.Ingredients.Bag from dfs.`complex.parquet` d;

select d.recipe, d.ingredients.bag from dfs.`complex.parquet` d;
select d.Recipe, d.ingredients.baG from dfs.`complex.parquet` d;

select d.ingredients.bag[0].name from dfs.`complex.parquet` d;
select d.ingredients.baG[0].name from dfs.`complex.parquet` d;
select d.ingredients.bag[0].naMe from dfs.`complex.parquet` d;
select d.ingredients.baG[0].naMe from dfs.`complex.parquet` d;


Thanks,

Parth Chandra


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message