spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-23515) JsonProtocol.sparkEventToJson can OOM when jsonifying an event
Date Mon, 26 Feb 2018 00:08:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376306#comment-16376306
] 

Apache Spark commented on SPARK-23515:
--------------------------------------

User 'brkyvz' has created a pull request for this issue:
https://github.com/apache/spark/pull/20673

> JsonProtocol.sparkEventToJson can OOM when jsonifying an event
> --------------------------------------------------------------
>
>                 Key: SPARK-23515
>                 URL: https://issues.apache.org/jira/browse/SPARK-23515
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.1
>            Reporter: Burak Yavuz
>            Assignee: Burak Yavuz
>            Priority: Major
>
> {code}
> def sparkEventToJson(event: SparkListenerEvent)
> {code}
> has a fallback method which creates a JSON object by turning an unrecognized event to
Json and then parsing it again. This method materializes the whole string to parse the json
record, which is unnecessary and can cause OOMs as seen in the stacktrace below:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3664)
> at java.lang.String.<init>(String.java:207)
> at java.lang.StringBuilder.toString(StringBuilder.java:407)
> at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:356)
> at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:235)
> at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:20)
> at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
> at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
> at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
> at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
> at org.json4s.jackson.JsonMethods$class.parse(JsonMethods.scala:20)
> at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:50)
> at org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:103){code}
>  
> We should just use the stream parsing to avoid such OOMs.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message