drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti" <venki.koruka...@gmail.com>
Subject Re: Review Request 30466: DRILL-133: LocalExchange (planning and parallelization)
Date Wed, 04 Feb 2015 16:27:47 GMT


> On Feb. 2, 2015, 10:18 p.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergingReceiverPOP.java,
line 58
> > <https://reviews.apache.org/r/30466/diff/1/?file=842680#file842680line58>
> >
> >     From these declarations, we can't tell what kind of List senders is; it might
be an ArrayList, or it might be a LinkedList. In the latter case, get(i) is expensive, because
it will iterate down the list to get to each item. Because of that, we should iterate over
that list instead, something like this:
> >     
> >     int i = 0;
> >     for(DrillbitEndpoint de : senders) {
> >       this.senders.put(i, de);
> >       ++i;
> >     }
> 
> Venki Korukanti wrote:
>     The reason why are using get(i) is because we want to create a mapping of "i" ->
DrillbitEndpoint. DrillbitEndpoint assigned to minor fragment 3 is at index 3 in the list.

Changed to use the iterator method you mentioned above as iterator is returning in order.


> On Feb. 2, 2015, 10:18 p.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnorderedReceiver.java,
line 62
> > <https://reviews.apache.org/r/30466/diff/1/?file=842687#file842687line62>
> >
> >     How does this get used? Would it be dangerous if the caller modified the map?
Should we return a copy?

Changed to return an immutable copy.


- Venki


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30466/#review70618
-----------------------------------------------------------


On Feb. 4, 2015, 2:24 a.m., Venki Korukanti wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30466/
> -----------------------------------------------------------
> 
> (Updated Feb. 4, 2015, 2:24 a.m.)
> 
> 
> Review request for drill, Chris Westin, Jacques Nadeau, and Steven Phillips.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> In this patch LocalExchange contains only multiplexing exchange. Currently working on
demultiplexing exchange (there are few failures in executions and currently debugging those).
Sending this patch to get initial feedback.
> 
> Brief overview of changes:
> 1. Traverse the PRel tree after all optimizations. Whenever a HashToRandomExchangePrel
is encountered insert a MuxExchange before HashToRandomExchangePrel and DemuxExchange after
HashToRandomExchangePrel.
> 2. Parallelization changes: 
>    i) Traverse the physical operator tree and divide it into Fragments.
>    ii) Based on the affinity of the sending Exchange, set parallelization dependencies
between fragments. 
>    iii) Start parallelizing from the leaf fragments (fragment that have no other fragments
depending on them for parallelization info). Stats collection include collecting parallelization
info (minWidth, maxWidth, affinityMap) and cost.
> 3. Change the Receiver to accept set of (minorFragmentId, DrillbitEndpoint) as sender
list. This also involved few changes in DataCollector.
> 4. Change SingleSender to accept custom minorFragmentId instead of default minorFragmentId
of 0
> 
> 
> Diffs
> -----
> 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java df31f74

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractExchange.java
73280ea 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScan.java
5d0d9bf 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractReceiver.java
f621a26 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractSender.java
53a0721 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Exchange.java 7be7f20

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 23860a3

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/HasAffinity.java 52462db

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Receiver.java 0c67770

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Sender.java bbd1b2c

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Store.java 94411ea

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastExchange.java
73a1d20 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastSender.java
1827367 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/DeMuxExchange.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashPartitionSender.java
bdb1362 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToMergeExchange.java
f62d922 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToRandomExchange.java
fac374b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergingReceiverPOP.java
f5dca1a 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MuxExchange.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionExchange.java
8e1526a 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionSender.java
0a2b9be 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/RangeSender.java
c8c8f43 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Screen.java 58c8e29

>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleMergeExchange.java
2914112 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleSender.java
4a11a51 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnionExchange.java
cfc21ac 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnorderedReceiver.java
3a4dd0e 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java
6db9f4a 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java
22fa047 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
f09acaa 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
4292c09 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Fragment.java ac63bde

>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/MakeFragmentsVisitor.java
8756e5b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java
961b603 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/ParallelizationInfo.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/PlanningSet.java
8cc6c85 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java
434cdd4 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Stats.java eda364b

>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/StatsCollector.java
41ff678 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 86b395e

>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DeMuxExchangePrel.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MuxExchangePrel.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
faa8546 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java
PRE-CREATION 
>   exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java
79603eb 
>   exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
f20627d 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java
aa1609d 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java
8335ed9 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java
5736df8 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockStorePOP.java 4c12d57

>   exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/SystemTableScan.java 053f5de

>   exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
c83106c 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/DataCollector.java dc016be

>   exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java
b0206f7 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/MergingCollector.java
ce14260 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/PartitionedCollector.java
5190d84 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java b33042b

>   exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestLocalExchange.java
PRE-CREATION 
>   exec/java-exec/src/test/java/org/apache/drill/exec/pop/PopUnitTestBase.java 9a32ff9

>   exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 6349b76

> 
> Diff: https://reviews.apache.org/r/30466/diff/
> 
> 
> Testing
> -------
> 
> Ran Functional and TPCH SF100 parquet verification tests.
> 
> 
> Thanks,
> 
> Venki Korukanti
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message