drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti" <venki.koruka...@gmail.com>
Subject Re: Review Request 30466: DRILL-133: LocalExchange (planning and parallelization)
Date Wed, 11 Feb 2015 19:26:42 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30466/
-----------------------------------------------------------

(Updated Feb. 11, 2015, 7:26 p.m.)


Review request for drill, Chris Westin, Jacques Nadeau, and Steven Phillips.


Changes
-------

Added ArrayWrapped Map to speedup get calls. 
Added more test coverage.


Repository: drill-git


Description
-------

In this patch LocalExchange contains only multiplexing exchange. Currently working on demultiplexing
exchange (there are few failures in executions and currently debugging those). Sending this
patch to get initial feedback.

Brief overview of changes:
1. Traverse the PRel tree after all optimizations. Whenever a HashToRandomExchangePrel is
encountered insert a MuxExchange before HashToRandomExchangePrel and DemuxExchange after HashToRandomExchangePrel.
2. Parallelization changes: 
   i) Traverse the physical operator tree and divide it into Fragments.
   ii) Based on the affinity of the sending Exchange, set parallelization dependencies between
fragments. 
   iii) Start parallelizing from the leaf fragments (fragment that have no other fragments
depending on them for parallelization info). Stats collection include collecting parallelization
info (minWidth, maxWidth, affinityMap) and cost.
3. Change the Receiver to accept set of (minorFragmentId, DrillbitEndpoint) as sender list.
This also involved few changes in DataCollector.
4. Change SingleSender to accept custom minorFragmentId instead of default minorFragmentId
of 0


Diffs (updated)
-----

  contrib/storage-hbase/src/test/java/org/apache/drill/hbase/BaseHBaseTest.java 1152b7b 
  contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/HiveTestBase.java 7e3b6c8

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java df31f74

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/MinorFragmentEndpoint.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractExchange.java 73280ea

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScan.java
5d0d9bf 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractReceiver.java f621a26

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractSender.java 53a0721

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Exchange.java 7be7f20 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperatorUtil.java
dfcb113 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Receiver.java 0c67770 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Sender.java bbd1b2c 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastExchange.java
73a1d20 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastSender.java
1827367 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/DeMuxExchange.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashPartitionSender.java
bdb1362 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToMergeExchange.java
f62d922 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToRandomExchange.java
fac374b 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergingReceiverPOP.java
f5dca1a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MuxExchange.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionExchange.java
8e1526a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionSender.java
0a2b9be 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/RangeSender.java c8c8f43

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Screen.java 58c8e29 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleMergeExchange.java
2914112 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleSender.java 4a11a51

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnionExchange.java cfc21ac

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnorderedReceiver.java
3a4dd0e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java
6db9f4a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java
22fa047 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java
d78ba8e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
f09acaa 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
4292c09 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java
52b892e 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Fragment.java ac63bde

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/MakeFragmentsVisitor.java
8756e5b 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java 961b603

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/ParallelizationInfo.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/PlanningSet.java 8cc6c85

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java
434cdd4 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Stats.java eda364b 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/StatsCollector.java
41ff678 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 86b395e

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DeMuxExchangePrel.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MuxExchangePrel.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
faa8546 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
6b3d301 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
aa0a5ad 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 9902443

  exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java aa1609d

  exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java
8335ed9 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java 5736df8

  exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/SystemTableScan.java 053f5de

  exec/java-exec/src/main/java/org/apache/drill/exec/util/ArrayWrappedIntIntMap.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
c83106c 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/MergingCollector.java ce14260

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/PartitionedCollector.java
5190d84 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a 
  exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java 17bcb79 
  exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java c52545d 
  exec/java-exec/src/test/java/org/apache/drill/exec/compile/TestClassTransformation.java
bb855c9 
  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestLocalExchange.java
PRE-CREATION 
  exec/java-exec/src/test/java/org/apache/drill/exec/pop/PopUnitTestBase.java 9a32ff9 
  exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 6349b76

  exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java
219e66f 
  exec/java-exec/src/test/java/org/apache/drill/exec/util/TestArrayWrappedIntIntMap.java PRE-CREATION


Diff: https://reviews.apache.org/r/30466/diff/


Testing
-------

Ran Functional and TPCH SF100 parquet verification tests.


Thanks,

Venki Korukanti


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message