spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang-Chi Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-28732) org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java' when storing the result of a count aggregation in an integer
Date Fri, 16 Aug 2019 21:20:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-28732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909409#comment-16909409
] 

Liang-Chi Hsieh commented on SPARK-28732:
-----------------------------------------

As {{count}} return type is LongType, I think it is reasonable that it can't be fit into an
Int column. The problem here might be the error is not friendly.

Normally, if we want to map dataset to specified type, an exception like this should be thrown,
if it is incompatible:
{code}
You can either add an explicit cast to the input data or choose a higher precision type of
the field in the target object;                                                          
                                 
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveUpCast$$fail(Analyzer.scala:2801)
                                                         
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$$anonfun$apply$32$$anonfun$applyOrElse$143.applyOrElse(Analyzer.scala:2821)
                                                                       
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$$anonfun$apply$32$$anonfun$applyOrElse$143.applyOrElse(Analyzer.scala:2812)
                                                                       
{code}

> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java' when storing the result
of a count aggregation in an integer
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-28732
>                 URL: https://issues.apache.org/jira/browse/SPARK-28732
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0
>            Reporter: Alix Métivier
>            Priority: Major
>
> I am using agg function on a dataset, and i want to count the number of lines upon grouping
columns. I would like to store the result of this count in an integer, but it fails with this
output : 
> {code}
> [ERROR]: org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to
compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 89, Column
53: No applicable constructor/method found for actual parameters "long"; candidates are: "java.lang.Integer(int)",
"java.lang.Integer(java.lang.String)"
> Here is the line 89 and a few others to understand :
> /* 085 */ long value13 = i.getLong(5);
>  /* 086 */ argValue4 = value13;
>  /* 087 */
>  /* 088 */
>  /* 089 */ final java.lang.Integer value12 = false ? null : new java.lang.Integer(argValue4);
> {code}
>  
> As per Integer documentation, there is not constructor for the type Long, so this is
why the generated code fails.
> Here is my code : 
> {code}
> org.apache.spark.sql.Dataset<row2Struct> ds_row2 = ds_conntAggregateRow_1_Out_1
>  .groupBy(org.apache.spark.sql.functions.col("n_name").as("n_nameN"),
>  org.apache.spark.sql.functions.col("o_year").as("o_yearN"))
>  .agg(org.apache.spark.sql.functions.count("n_name").as("countN"),
>  .as(org.apache.spark.sql.Encoders.bean(row2Struct.class));
> {code}
> row2Struct class is composed of n_nameN: String, o_yearN: String, countN: Int
> If countN is a Long, code above wont fail
> If it is an Int, it works in 1.6 and 2.0, but fails on version 2.1+
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message