drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti" <venki.koruka...@gmail.com>
Subject Re: Review Request 30466: DRILL-133: LocalExchange (planning and parallelization)
Date Wed, 04 Feb 2015 02:24:48 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30466/
-----------------------------------------------------------

(Updated Feb. 4, 2015, 2:24 a.m.)


Review request for drill, Chris Westin, Jacques Nadeau, and Steven Phillips.


Changes
-------

Added demux and new tests. Also addressed some review comments.


Repository: drill-git


Description
-------

In this patch LocalExchange contains only multiplexing exchange. Currently working on demultiplexing
exchange (there are few failures in executions and currently debugging those). Sending this
patch to get initial feedback.

Brief overview of changes:
1. Traverse the PRel tree after all optimizations. Whenever a HashToRandomExchangePrel is
encountered insert a MuxExchange before HashToRandomExchangePrel and DemuxExchange after HashToRandomExchangePrel.
2. Parallelization changes: 
   i) Traverse the physical operator tree and divide it into Fragments.
   ii) Based on the affinity of the sending Exchange, set parallelization dependencies between
fragments. 
   iii) Start parallelizing from the leaf fragments (fragment that have no other fragments
depending on them for parallelization info). Stats collection include collecting parallelization
info (minWidth, maxWidth, affinityMap) and cost.
3. Change the Receiver to accept set of (minorFragmentId, DrillbitEndpoint) as sender list.
This also involved few changes in DataCollector.
4. Change SingleSender to accept custom minorFragmentId instead of default minorFragmentId
of 0


Diffs (updated)
-----

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java df31f74

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractExchange.java 73280ea

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScan.java
5d0d9bf 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractReceiver.java f621a26

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractSender.java 53a0721

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Exchange.java 7be7f20 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 23860a3

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/HasAffinity.java 52462db

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Receiver.java 0c67770 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Sender.java bbd1b2c 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Store.java 94411ea 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastExchange.java
73a1d20 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastSender.java
1827367 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/DeMuxExchange.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashPartitionSender.java
bdb1362 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToMergeExchange.java
f62d922 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToRandomExchange.java
fac374b 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergingReceiverPOP.java
f5dca1a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MuxExchange.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionExchange.java
8e1526a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionSender.java
0a2b9be 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/RangeSender.java c8c8f43

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Screen.java 58c8e29 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleMergeExchange.java
2914112 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleSender.java 4a11a51

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnionExchange.java cfc21ac

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnorderedReceiver.java
3a4dd0e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java
6db9f4a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java
22fa047 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
f09acaa 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
4292c09 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Fragment.java ac63bde

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/MakeFragmentsVisitor.java
8756e5b 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java 961b603

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/ParallelizationInfo.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/PlanningSet.java 8cc6c85

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java
434cdd4 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Stats.java eda364b 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/StatsCollector.java
41ff678 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 86b395e

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DeMuxExchangePrel.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MuxExchangePrel.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
faa8546 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
79603eb 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
f20627d 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java aa1609d

  exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java
8335ed9 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java 5736df8

  exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockStorePOP.java 4c12d57

  exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/SystemTableScan.java 053f5de

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
c83106c 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/DataCollector.java dc016be

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java b0206f7

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/MergingCollector.java ce14260

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/PartitionedCollector.java
5190d84 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java b33042b 
  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestLocalExchange.java
PRE-CREATION 
  exec/java-exec/src/test/java/org/apache/drill/exec/pop/PopUnitTestBase.java 9a32ff9 
  exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 6349b76


Diff: https://reviews.apache.org/r/30466/diff/


Testing
-------

Ran Functional and TPCH SF100 parquet verification tests.


Thanks,

Venki Korukanti


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message