spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang-Chi Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-6586) Add the capability of retrieving original logical plan of DataFrame
Date Sun, 29 Mar 2015 23:24:53 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386009#comment-14386009
] 

Liang-Chi Hsieh edited comment on SPARK-6586 at 3/29/15 11:24 PM:
------------------------------------------------------------------

Not true. Because DataFrame now is given analyzed plan after its many operations, {{df.queryExecution.logical}}
is analyzed plan instead of the original logical plan.

You can check the pr #5217 for the modification.


was (Author: viirya):
Not true. Because DataFrame now is given analyzed plan after its many operations, {{df.queryExecution.logical}}
is analyzed plan instead of the original logical plan.

> Add the capability of retrieving original logical plan of DataFrame
> -------------------------------------------------------------------
>
>                 Key: SPARK-6586
>                 URL: https://issues.apache.org/jira/browse/SPARK-6586
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Liang-Chi Hsieh
>            Priority: Minor
>
> In order to solve a bug, since #5217, {{DataFrame}} now uses analyzed plan instead of
logical plan. However, by doing that we can't know the logical plan of a {{DataFrame}}. But
it might be still useful and important to retrieve the original logical plan in some use cases.
> In this pr, we introduce the capability of retrieving original logical plan of {{DataFrame}}.
> The approach is that we add an {{analyzed}} variable to {{LogicalPlan}}. Once {{Analyzer}}
finishes analysis, it sets {{analyzed}} of {{LogicalPlan}} as {{true}}.  In {{QueryExecution}},
we keep the original logical plan in the analyzed plan. In {{LogicalPlan}}, a method {{originalPlan}}
is added to recursively replace the analyzed logical plan with original logical plan and retrieve
it.
> Besides the capability of retrieving original logical plan, this modification also can
avoid do plan analysis if it is already analyzed.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message