spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kunkel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21214) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '
Date Mon, 26 Jun 2017 17:29:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063461#comment-16063461
] 

Michael Kunkel commented on SPARK-21214:
----------------------------------------

Greetings,

Would you please inform me on the location of the mailing list?

BR
MK
----------------------------------------
Michael C. Kunkel, USMC, PhD
Forschungszentrum J├╝lich
Nuclear Physics Institute and Juelich Center for Hadron Physics
Experimental Hadron Structure (IKP-1)
www.fz-juelich.de/ikp




------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------



> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-21214
>                 URL: https://issues.apache.org/jira/browse/SPARK-21214
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API, Spark Core, SQL
>    Affects Versions: 2.1.1
>         Environment: macOSX
>            Reporter: Michael Kunkel
>
> First Spark project.
> I have a Java method that returns a Dataset<Row>. I want to convert this to a Dataset<Object>,
where the Object is named StatusChangeDB. I have created a POJO StatusChangeDB.java and coded
it with all the query objects found in the mySQL table.
> I then create a Encoder and convert the Dataset<Row> to a Dataset<StatusChangeDB>.
However when I try to .show() the values of the Dataset<StatusChangeDB> I receive the
error
> bq. 
> bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
'`hvpinid_quad`' given input columns: [status_change_type, superLayer, loclayer, sector, locwire];
> bq. 	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> bq. 	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86)
> bq. 	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290)
> bq. 	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324)
> bq. 	at scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246)
> bq. 	at scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246)
> bq. 	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> bq. 	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> bq. 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> bq. 	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> bq. 	at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311)
> bq. 	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
> bq. 	at scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25)
> bq. 	at scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88)
> bq. 	at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:266)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:276)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$6.apply(QueryPlan.scala:285)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:285)
> bq. 	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:255)
> bq. 	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:83)
> bq. 	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:76)
> bq. 	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:128)
> bq. 	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:76)
> bq. 	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:57)
> bq. 	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.resolveAndBind(ExpressionEncoder.scala:259)
> bq. 	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:209)
> bq. 	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167)
> bq. 	at org.apache.spark.sql.Dataset$.apply(Dataset.scala:58)
> bq. 	at org.apache.spark.sql.Dataset.as(Dataset.scala:376)
> I do not know how to have whomever replicate this, but here are some methods that are
used on my end. I hope the error can be seen from the following codes:
> 	public static Dataset<Row> mySqlDataset() {
> 		SparkSession spSession = getSession();
> 		spSession.sql("set spark.sql.caseSensitive=false");
> 		 Dataset<Row> demoDf = spSession.read().format("jdbc").options(jdbcOptions()).load();
> }
> where jdbcOptions() are
> 	public static Map<String, String> jdbcOptions() {
> 		Map<String, String> jdbcOptions = new HashMap<String, String>();
> 		jdbcOptions.put("url", "jdbc:mysql://localhost:3306/test");
> 		jdbcOptions.put("driver", "com.mysql.jdbc.Driver");
> 		jdbcOptions.put("dbtable", "status_change");
> 		jdbcOptions.put("user", "root");
> 		jdbcOptions.put("password", "");
> 		return jdbcOptions;
> 	}
> the method that fails is
> 	public Dataset<StatusChangeDB> compareRunII(String str) {
> 		Dataset<Row> tempDF = SparkManager.mySqlDataset()
> 				.select("loclayer", "superLayer", "sector", "locwire", "status_change_type")
> 				.filter(col("runno").equalTo(str));
> 		return tempDF.as(SparkManager.statusChangeDBEncoder());
> 	}
> where SparkManager.statusChangeDBEncoder() is 
> 	public static Encoder<StatusChangeDB> statusChangeDBEncoder() {
> 		return Encoders.bean(StatusChangeDB.class);
> 	}
> and StatusChangeDB is just a POJO that works because I am able to create Dataset<StatusChangeDB>
from a datafile.
> There is no help on this issue on Google or here for the JavaAPI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message