spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22222) Fix the ARRAY_MAX in BufferHolder and add a test
Date Wed, 08 Nov 2017 18:12:01 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244455#comment-16244455
] 

Apache Spark commented on SPARK-22222:
--------------------------------------

User 'dongjoon-hyun' has created a pull request for this issue:
https://github.com/apache/spark/pull/19697

> Fix the ARRAY_MAX in BufferHolder and add a test
> ------------------------------------------------
>
>                 Key: SPARK-22222
>                 URL: https://issues.apache.org/jira/browse/SPARK-22222
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: Feng Liu
>            Assignee: Feng Liu
>             Fix For: 2.3.0
>
>
> This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to `Int.MaxValue
- 8`. It is not a valid number because it will cause the following line fail when such a large
byte array is allocated: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86
We need to make sure the new length is a multiple of 8.
> We need to add one test for the fix. Note that the test should work independently with
the heap size of the test JVM. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message