drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Gilmore" <dragoncu...@gmail.com>
Subject Review Request 33836: DRILL-1950: Parquet pushdown filtering
Date Tue, 05 May 2015 01:27:10 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33836/
-----------------------------------------------------------

Review request for drill and Jacques Nadeau.


Repository: drill-git


Description
-------

An implementation of Parquet pushdown filtering for Drill.  More details can be found in the
JIRA item (DRILL-1950).


Diffs
-----

  contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java
c10b0abd0d79e94524a7f4ce0043cd63ace60a69 
  contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java
f4baf3b023f83452eea2fccc5b54feeaa262b53c 
  contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoStoragePlugin.java
e46d8ec589785d5db1b655dae0857464fa3c9159 
  exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java bd93206d531699ee4f3fcaff11ce0df143f3032f

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java ca2a0486d7f18fb27a592c0824dbef1a31cb3b70

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b98778d2a90eefc297df81739ce945beb1080021

  exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
1a8559e54810bd9c678bece13b1f42aad89c00e2 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce8763d92e58993b78b48993444cb84e5c3

  exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c8b1b9c91d4d6a873ff798ae096b0bd1e7

  exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6a8446b02610ccb0e9f2b4a667142b3900

  exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java c5ca41b160782e6421c6d9f90dbfea555d5d14b8

  exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 58d5b426937accee0c3ba1752e40bfec8df23e19

  exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java
6e1e0cc25109beabd119fbb6aac13a3fea670c55 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaStoragePlugin.java
77c6b9ad0c53fc29e7ed73d9d184f411e3f21308 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/EmptyRowGroupScan.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/EmptyScanBatchCreator.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetCompareFunctionProcessor.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
e204a2c28a39c3a647686f09445af3682185fa5e 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java acac61f35448b2884c60a03830532ccd24bc638a

  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetPushDownFilter.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRowGroupScan.java
fd40f417a65921efccae2deb5226e6adf86364c3 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetScanBatchCreator.java
c1f815e68945bd454d0b6fd268a17a33b6eda00c 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java
9f8357bef94d6bc00392d32a61bf35ca10a8feaa 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetRecordMaterializer.java
720e8be7f5befb3039dccd062230ca686cb5f32e 
  exec/java-exec/src/main/java/parquet/hadoop/FilterPredicateSerializer.java PRE-CREATION

  exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestFilterPushdown.java
PRE-CREATION 
  exec/java-exec/src/test/java/parquet/hadoop/TestFilterPredicateSerializer.java PRE-CREATION

  exec/java-exec/src/test/resources/parquet/pushdown/0_0_0.parquet PRE-CREATION 
  exec/java-exec/src/test/resources/parquet/pushdown/0_0_1.parquet PRE-CREATION 
  protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ac1bcbb8761b69dc944180234399c0badc517e4f


Diff: https://reviews.apache.org/r/33836/diff/


Testing
-------

I have created a number of test cases to test that the filter is correctly pushed down in
various scenarios.  This also ensures that the pushdown filter is correctly working as it
must run through the row group filtering to estimate fewer rows scanned and for the optimizer
to pick that as a better plan.


Thanks,

Adam Gilmore


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message