spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saurabh Santhosh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-11633) HiveContext throws TreeNode Exception : Failed to Copy Node
Date Tue, 10 Nov 2015 17:55:11 GMT
Saurabh Santhosh created SPARK-11633:
----------------------------------------

             Summary: HiveContext throws TreeNode Exception : Failed to Copy Node
                 Key: SPARK-11633
                 URL: https://issues.apache.org/jira/browse/SPARK-11633
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.4.1
            Reporter: Saurabh Santhosh
            Priority: Critical


h2. HiveContext#sql is throwing the following exception in a specific scenario :

h2. Exception :

Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: 
Failed to copy node.
Is otherCopyArgs specified correctly for LogicalRDD.
Exception message: wrong number of arguments
ctor: public org.apache.spark.sql.execution.LogicalRDD
(scala.collection.Seq,org.apache.spark.rdd.RDD,org.apache.spark.sql.SQLContext)?

h2. Code :

{code:title=SparkClient.java|borderStyle=solid}
StructField[] fields = new StructField[2];
fields[0] = new StructField("F1", DataTypes.StringType, true, Metadata.empty());
fields[1] = new StructField("F2", DataTypes.StringType, true, Metadata.empty());
    
JavaRDD<Row> rdd = javaSparkContext.parallelize(Arrays.asList(RowFactory.create("",
"", 0)));

DataFrame df = sparkHiveContext.createDataFrame(rdd, new StructType(fields));
sparkHiveContext.registerDataFrameAsTable(df, "t1");

DataFrame aliasedDf = sparkHiveContext.sql("select f1, F2 as F2 from t1");

sparkHiveContext.registerDataFrameAsTable(aliasedDf, "t2");
sparkHiveContext.registerDataFrameAsTable(aliasedDf, "t3");

sparkHiveContext.sql("select a.F1 from t2 a inner join t3 b on a.F2=b.F2");

{code}

h2. Observations :

* if F1(exact name of field) is used instead of f1, the code works correctly.
* If alias is not used for F2, then also code works irrespective of case of F1.
* if Field F2 is not used in the final query also the code works correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message