spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Barclay (JIRA)" <>
Subject [jira] [Commented] (SPARK-16652) JVM crash from unsafe memory access for Dataset of class with List[Long]
Date Tue, 26 Jul 2016 17:19:20 GMT


Daniel Barclay commented on SPARK-16652:

Not that I can really tell either, but this does not seem to be generated code that's invalid
in the sense of not following the class file format or failing bytecode validation.

It just seems to be invalid in the sense of accessing the wrong memory (e.g., something accessed
the wrong address, or allocation/deallocation got out of sync):

1. Recall that I initially saw "fault occurred in a recent unsafe memory access" exceptions
(before I trimmed the test case down to its current form).

2. *Also, note how changing the string literal on line 63(?) in my test case changes things:
- With the current multi-character value, the program crashes.
- With a two-character value, it also crashes.
- With a one-character value, it runs.
- With a zero-character value (empty string), it runs.
- With a null value, it runs.

(When I was getting the "fault ... in ... unsafe" messages, a one-character value would crash,
while an empty string or null would run.)

> JVM crash from unsafe memory access for Dataset of class with List[Long]
> ------------------------------------------------------------------------
>                 Key: SPARK-16652
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.6.1, 1.6.2
>         Environment: Scala 2.10.<various>
> JDK: "Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)"
> MacOs 10.11.2
>            Reporter: Daniel Barclay
>         Attachments: UnsafeAccessCrashBugTest.scala
> Generating and writing out a {{Dataset}} of a class that has a {{List}} (at least if
it's {{List\[Long]}}) member and a {{String}} member causes a JVM crash.
> The crash seems to be related to unsafe memory access especially because earlier code
(before I got it reduced to the current bug test case)  reported "{{java.lang.InternalError:
a fault occurred in a recent unsafe memory access operation in compiled Java code}}".

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message