Return-Path: X-Original-To: apmail-drill-dev-archive@www.apache.org Delivered-To: apmail-drill-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5023B107C8 for ; Mon, 9 Feb 2015 15:03:46 +0000 (UTC) Received: (qmail 5088 invoked by uid 500); 9 Feb 2015 15:03:46 -0000 Delivered-To: apmail-drill-dev-archive@drill.apache.org Received: (qmail 5028 invoked by uid 500); 9 Feb 2015 15:03:46 -0000 Mailing-List: contact dev-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list dev@drill.apache.org Received: (qmail 5010 invoked by uid 500); 9 Feb 2015 15:03:45 -0000 Delivered-To: apmail-incubator-drill-dev@incubator.apache.org Received: (qmail 5002 invoked by uid 99); 9 Feb 2015 15:03:45 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Feb 2015 15:03:45 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 5759B1CE09E; Mon, 9 Feb 2015 15:03:45 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2798139575243966737==" MIME-Version: 1.0 Subject: Re: Review Request 30466: DRILL-133: LocalExchange (planning and parallelization) From: "Venki Korukanti" To: "Jacques Nadeau" , "Chris Westin" , "Steven Phillips" Cc: "Venki Korukanti" , "drill" Date: Mon, 09 Feb 2015 15:03:45 -0000 Message-ID: <20150209150345.29075.60371@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Venki Korukanti" X-ReviewGroup: drill-git X-ReviewRequest-URL: https://reviews.apache.org/r/30466/ X-Sender: "Venki Korukanti" References: <20150204233738.1286.80989@reviews.apache.org> In-Reply-To: <20150204233738.1286.80989@reviews.apache.org> Reply-To: "Venki Korukanti" X-ReviewRequest-Repository: drill-git --===============2798139575243966737== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit > On Feb. 4, 2015, 11:37 p.m., Chris Westin wrote: > > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java, line 257 > > > > > > Ok. If we use a custom Comparator here, as I suggested above in the comments about compareTo() (which looks very weird with only some members, which is why I suggested removing it), then we can reverse the ordering in the Comparator, and skip the reverse(). Updated to use custom Comparator here and remove Comparable implementation from EndpointAffinity. > On Feb. 4, 2015, 11:37 p.m., Chris Westin wrote: > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java, line 45 > > > > > > Great. In that case, endpoint should be final. Already converted to final in v2 patch. - Venki ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30466/#review71077 ----------------------------------------------------------- On Feb. 4, 2015, 4:30 p.m., Venki Korukanti wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30466/ > ----------------------------------------------------------- > > (Updated Feb. 4, 2015, 4:30 p.m.) > > > Review request for drill, Chris Westin, Jacques Nadeau, and Steven Phillips. > > > Repository: drill-git > > > Description > ------- > > In this patch LocalExchange contains only multiplexing exchange. Currently working on demultiplexing exchange (there are few failures in executions and currently debugging those). Sending this patch to get initial feedback. > > Brief overview of changes: > 1. Traverse the PRel tree after all optimizations. Whenever a HashToRandomExchangePrel is encountered insert a MuxExchange before HashToRandomExchangePrel and DemuxExchange after HashToRandomExchangePrel. > 2. Parallelization changes: > i) Traverse the physical operator tree and divide it into Fragments. > ii) Based on the affinity of the sending Exchange, set parallelization dependencies between fragments. > iii) Start parallelizing from the leaf fragments (fragment that have no other fragments depending on them for parallelization info). Stats collection include collecting parallelization info (minWidth, maxWidth, affinityMap) and cost. > 3. Change the Receiver to accept set of (minorFragmentId, DrillbitEndpoint) as sender list. This also involved few changes in DataCollector. > 4. Change SingleSender to accept custom minorFragmentId instead of default minorFragmentId of 0 > > > Diffs > ----- > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java df31f74 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractExchange.java 73280ea > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScan.java 5d0d9bf > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractReceiver.java f621a26 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractSender.java 53a0721 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Exchange.java 7be7f20 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 23860a3 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/HasAffinity.java 52462db > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperatorUtil.java dfcb113 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Receiver.java 0c67770 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Sender.java bbd1b2c > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Store.java 94411ea > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastExchange.java 73a1d20 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastSender.java 1827367 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/DeMuxExchange.java PRE-CREATION > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashPartitionSender.java bdb1362 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToMergeExchange.java f62d922 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToRandomExchange.java fac374b > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergingReceiverPOP.java f5dca1a > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MuxExchange.java PRE-CREATION > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionExchange.java 8e1526a > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionSender.java 0a2b9be > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/RangeSender.java c8c8f43 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Screen.java 58c8e29 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleMergeExchange.java 2914112 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleSender.java 4a11a51 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnionExchange.java cfc21ac > exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnorderedReceiver.java 3a4dd0e > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 6db9f4a > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java 22fa047 > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09 > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Fragment.java ac63bde > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/MakeFragmentsVisitor.java 8756e5b > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java 961b603 > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/ParallelizationInfo.java PRE-CREATION > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/PlanningSet.java 8cc6c85 > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java 434cdd4 > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Stats.java eda364b > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/StatsCollector.java 41ff678 > exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 86b395e > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DeMuxExchangePrel.java PRE-CREATION > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MuxExchangePrel.java PRE-CREATION > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java faa8546 > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java PRE-CREATION > exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java 79603eb > exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java f20627d > exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java aa1609d > exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java 8335ed9 > exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java 5736df8 > exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockStorePOP.java 4c12d57 > exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/SystemTableScan.java 053f5de > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java c83106c > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/DataCollector.java dc016be > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/IncomingBuffers.java b0206f7 > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/MergingCollector.java ce14260 > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/PartitionedCollector.java 5190d84 > exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java b33042b > exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestLocalExchange.java PRE-CREATION > exec/java-exec/src/test/java/org/apache/drill/exec/pop/PopUnitTestBase.java 9a32ff9 > exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 6349b76 > > Diff: https://reviews.apache.org/r/30466/diff/ > > > Testing > ------- > > Ran Functional and TPCH SF100 parquet verification tests. > > > Thanks, > > Venki Korukanti > > --===============2798139575243966737==--