spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-16898) Adds argument type information for typed logical plan like MapElements, TypedFilter, and AppendColumn
Date Thu, 04 Aug 2016 16:24:20 GMT

     [ https://issues.apache.org/jira/browse/SPARK-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-16898:
------------------------------------

    Assignee: Apache Spark

> Adds argument type information for typed logical plan like MapElements, TypedFilter,
and AppendColumn
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16898
>                 URL: https://issues.apache.org/jira/browse/SPARK-16898
>             Project: Spark
>          Issue Type: Bug
>            Reporter: Sean Zhong
>            Assignee: Apache Spark
>            Priority: Minor
>
> Typed logical plan like MapElements, TypedFilter, and AppendColumn contains a closure
field: {{func: (T) => Boolean}}. For example class TypedFilter's signature is:
> {code}
> case class TypedFilter(
>     func: AnyRef,
>     deserializer: Expression,
>     child: LogicalPlan) extends UnaryNode
> {code} 
> From the above class signature, we cannot easily find:
> 1. What is the input argument's type of the closure {{func}}? How do we know which apply
method to pick if there are multiple overloaded apply methods?
> 2. What is the input argument's schema? 
> With this info, it is easier for us to define some custom optimizer rule to translate
these typed logical plan to more efficient implementation, like the closure optimization idea
in SPARK-14083.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message