spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Corey J. Nolet (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-5296) Predicate Pushdown (BaseRelation) to have an interface that will accept OR filters
Date Fri, 13 Feb 2015 17:34:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320417#comment-14320417
] 

Corey J. Nolet edited comment on SPARK-5296 at 2/13/15 5:33 PM:
----------------------------------------------------------------

bq.  I would only pass down ORs though, as we already split up ANDs and filter/pass them down
already implicitly.

I could understand wanting to treat the Array as a conjunction but how would you support arbitrarily
nested AND and OR filters if you don't have AND as part of the tree? This is really what interests
me about the CatalystScan- I can unroll the parts of the tree that i am able to support and
those can be arbitrarily nested. 


was (Author: sonixbp):
bq.  I would only pass down ORs though, as we already split up ANDs and filter/pass them down
already implicitly.

I could understand wanting to treat the Array as a conjunction but how would you support arbitrarily
nested AND and OR filters if you don't have AND as part of the tree?

> Predicate Pushdown (BaseRelation) to have an interface that will accept OR filters
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-5296
>                 URL: https://issues.apache.org/jira/browse/SPARK-5296
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Corey J. Nolet
>            Assignee: Cheng Lian
>            Priority: Critical
>
> Currently, the BaseRelation API allows a FilteredRelation to handle an Array[Filter]
which represents filter expressions that are applied as an AND operator.
> We should support OR operations in a BaseRelation as well. I'm not sure what this would
look like in terms of API changes, but it almost seems like a FilteredUnionedScan BaseRelation
(the name stinks but you get the idea) would be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message