spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] cloud-fan edited a comment on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
Date Tue, 10 Mar 2020 06:01:44 GMT
cloud-fan edited a comment on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for
decimal aggregate sum by returning null on decimal overflow
URL: https://github.com/apache/spark/pull/27627#issuecomment-596918804
 
 
   @skambha thanks for your investigation! Very helpful.
   
   I'm trying to determine the scope of this bug. Your analysis is for global aggregate (no
GROUP BY), when we only write out the sum value at the end of partial aggregate. I tried with
group by
   ```
   df.select(expr(s"cast('$decimalStr' as decimal (38, 18)) as d"), lit(1).as("key"))
   .groupBy("key").agg(sum($"d")).show
   java.lang.ArithmeticException: Decimal precision 39 exceeds max precision 38
   	at org.apache.spark.sql.types.Decimal.set(Decimal.scala:122)
   	at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:574)
   	at org.apache.spark.sql.types.Decimal.apply(Decimal.scala)
   	at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:393)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doConsume_0$(Unknown
Source)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithKeys_1$(Unknown
Source)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithKeys_0$(Unknown
Source)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
   ...
   ```
   
   For aggregate with GROUP BY, we write out the sum value to unsafe row(the aggregate hash
map) at each step. Do you what happens there? I'd expect we write overflowed decimal as null
to the aggregate hash map, but it seems not the case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message