spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saurabh Santhosh (JIRA)" <>
Subject [jira] [Created] (SPARK-11633) HiveContext throws TreeNode Exception : Failed to Copy Node
Date Tue, 10 Nov 2015 17:55:11 GMT
Saurabh Santhosh created SPARK-11633:

             Summary: HiveContext throws TreeNode Exception : Failed to Copy Node
                 Key: SPARK-11633
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.4.1
            Reporter: Saurabh Santhosh
            Priority: Critical

h2. HiveContext#sql is throwing the following exception in a specific scenario :

h2. Exception :

Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: 
Failed to copy node.
Is otherCopyArgs specified correctly for LogicalRDD.
Exception message: wrong number of arguments
ctor: public org.apache.spark.sql.execution.LogicalRDD

h2. Code :

StructField[] fields = new StructField[2];
fields[0] = new StructField("F1", DataTypes.StringType, true, Metadata.empty());
fields[1] = new StructField("F2", DataTypes.StringType, true, Metadata.empty());
JavaRDD<Row> rdd = javaSparkContext.parallelize(Arrays.asList(RowFactory.create("",
"", 0)));

DataFrame df = sparkHiveContext.createDataFrame(rdd, new StructType(fields));
sparkHiveContext.registerDataFrameAsTable(df, "t1");

DataFrame aliasedDf = sparkHiveContext.sql("select f1, F2 as F2 from t1");

sparkHiveContext.registerDataFrameAsTable(aliasedDf, "t2");
sparkHiveContext.registerDataFrameAsTable(aliasedDf, "t3");

sparkHiveContext.sql("select a.F1 from t2 a inner join t3 b on a.F2=b.F2");


h2. Observations :

* if F1(exact name of field) is used instead of f1, the code works correctly.
* If alias is not used for F2, then also code works irrespective of case of F1.
* if Field F2 is not used in the final query also the code works correctly.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message