spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame
Date Mon, 22 Aug 2016 14:51:22 GMT

    [ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430907#comment-15430907
] 

Wenchen Fan commented on SPARK-14948:
-------------------------------------

actually `registerDataFrameAsTable` registers the dataframe as a temp table, see the document
of this method:
{code}
  /**
   * Registers the given [[DataFrame]] as a temporary table in the catalog. Temporary tables
exist
   * only during the lifetime of this instance of SQLContext.
   */
  private[sql] def registerDataFrameAsTable(df: DataFrame, tableName: String): Unit = {
    catalog.registerTable(TableIdentifier(tableName), df.logicalPlan)
  }
{code}

> Exception when joining DataFrames derived form the same DataFrame
> -----------------------------------------------------------------
>
>                 Key: SPARK-14948
>                 URL: https://issues.apache.org/jira/browse/SPARK-14948
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: Saurabh Santhosh
>
> h2. Spark Analyser is throwing the following exception in a specific scenario :
> h2. Exception :
> org.apache.spark.sql.AnalysisException: resolved attribute(s) F1#3 missing from asd#5,F2#4,F1#6,F2#7
in operator !Project [asd#5,F1#3];
> 	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:38)
> h2. Code :
> {code:title=SparkClient.java|borderStyle=solid}
>     StructField[] fields = new StructField[2];
>     fields[0] = new StructField("F1", DataTypes.StringType, true, Metadata.empty());
>     fields[1] = new StructField("F2", DataTypes.StringType, true, Metadata.empty());
>     JavaRDD<Row> rdd =
>         sparkClient.getJavaSparkContext().parallelize(Arrays.asList(RowFactory.create("a",
"b")));
>     DataFrame df = sparkClient.getSparkHiveContext().createDataFrame(rdd, new StructType(fields));
>     sparkClient.getSparkHiveContext().registerDataFrameAsTable(df, "t1");
>     DataFrame aliasedDf = sparkClient.getSparkHiveContext().sql("select F1 as asd, F2
from t1");
>     sparkClient.getSparkHiveContext().registerDataFrameAsTable(aliasedDf, "t2");
>     sparkClient.getSparkHiveContext().registerDataFrameAsTable(df, "t3");
>     
>     DataFrame join = aliasedDf.join(df, aliasedDf.col("F2").equalTo(df.col("F2")), "inner");
>     DataFrame select = join.select(aliasedDf.col("asd"), df.col("F1"));
>     select.collect();
> {code}
> h2. Observations :
> * This issue is related to the Data Type of Fields of the initial Data Frame.(If the
Data Type is not String, it will work.)
> * It works fine if the data frame is registered as a temporary table and an sql (select
a.asd,b.F1 from t2 a inner join t3 b on a.F2=b.F2) is written.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message