phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-1580) Support UNION ALL
Date Mon, 30 Mar 2015 23:45:53 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387622#comment-14387622
] 

James Taylor commented on PHOENIX-1580:
---------------------------------------

Thanks for the revisions, [~ayingshu]. Here's some feedback:
- You have to push down the ORDER BY and LIMIT to the individual scans in order for UNION
ALL to scale. It's very important to understand conceptually what is going on here. When you
have a set of iterators that are already in sorted order, you can do a merge sort between
them. Merge sorts are good because they don't require much memory and scale well for big data.
If you had to order all the rows from all the scans on the client, you'd essentially need
to buffer all of them on the client which is inherently a bad idea. The key here is how you
combine the set of individual iterators. Instead of using a ConcatResultIterator, which just
concatenates the scan results together, you need to do a merge sort between them using the
MergeSortTopNResultIterator. Something like this:
{code}
+    public final ResultIterator iterator(final List<? extends SQLCloseable> dependencies)
throws SQLException {
+        ResultIterator scanner;      
+
+        List<PeekingResultIterator> pIterators = new ArrayList<PeekingResultIterator>();
+        for (QueryPlan plan : this.getPlans()) {
+            pIterators.add(LookAheadResultIterator.wrap(plan.iterator()));
+        }
+
+        if (!orderBy.getOrderByExpressions().isEmpty()) { // TopN
+            scanner = new MergeSortTopNResultIterator(pIterators, limit, orderBy.getOrderByExpressions());
+        } else {
+            scanner = new ConcatResultIterator(pIterators);
+            if (limit != null) {
+                scanner = new LimitingResultIterator(scanner, limit);
+            }
+        }
+
+        return scanner;
+    }
+
{code}
- Don't make the WildcardParseNode constructor public, but instead go through the static factory
method.
{code}
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java b/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
index 9922c3f..ddf6f3b 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
@@ -37,7 +37,7 @@ public class WildcardParseNode extends TerminalParseNode {
 
     private final boolean isRewrite;
 
-    private WildcardParseNode(boolean isRewrite) {
+    public WildcardParseNode(boolean isRewrite) {
         this.isRewrite = isRewrite;
     }
{code}
- Similarly with ConcatResultIterator, don't make its constructor public, instead go through
the static factory method:
{code}
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
b/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
index fcc88aa..b3cf83b 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
@@ -40,7 +40,7 @@ public class ConcatResultIterator implements PeekingResultIterator {
         this.resultIterators = iterators;
     }
     
-    private ConcatResultIterator(List<PeekingResultIterator> iterators) {
+    public ConcatResultIterator(List<PeekingResultIterator> iterators) {
         this.resultIterators = null;
         this.iterators = iterators;
     }
{code}

> Support UNION ALL
> -----------------
>
>                 Key: PHOENIX-1580
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1580
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Alicia Ying Shu
>            Assignee: Alicia Ying Shu
>         Attachments: PHOENIX-1580-grammar.patch, Phoenix-1580-v1.patch, phoenix-1580-v1-wipe.patch,
phoenix-1580.patch, unionall-wipe.patch
>
>
> Select * from T1
> UNION ALL
> Select * from T2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message