Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 30 Sep 2017 04:38:04 +0000 (UTC)
From: "Sean Owen (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13105913.1506642936000.244039.1506746284484@Atlassian.JIRA>
In-Reply-To: <JIRA.13105913.1506642936000@Atlassian.JIRA>
References: <JIRA.13105913.1506642936000@Atlassian.JIRA> <JIRA.13105913.1506642936411@jira-lw-us.apache.org>
Subject: [jira] [Closed] (SPARK-22163) Design Issue of Spark Streaming that
 Causes Random Run-time Exception
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 30 Sep 2017 04:38:12 -0000


     [ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen closed SPARK-22163.
-----------------------------

> Design Issue of Spark Streaming that Causes Random Run-time Exception
> ---------------------------------------------------------------------
>
>                 Key: SPARK-22163
>                 URL: https://issues.apache.org/jira/browse/SPARK-22163
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams, Structured Streaming
>    Affects Versions: 2.2.0
>         Environment: Spark Streaming
> Kafka
> Linux
>            Reporter: Michael N
>            Priority: Critical
>
> The application objects can contain List and can be modified dynamically as well.   However, Spark Streaming framework asynchronously serializes the application's objects as the application runs.  Therefore, it causes random run-time exception on the List when Spark Streaming framework happens to serializes the application's objects while the application modifies a List in its own object.  
> In fact, there are multiple bugs reported about
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList.writeObject
> that are permutation of the same root cause. So the design issue of Spark streaming framework is that it should do this serialization asynchronously.  Instead, it should either
> 1. do this serialization synchronously. This is preferred to eliminate the issue completely.  Or
> 2. Allow it to be configured per application whether to do this serialization synchronously or asynchronously, depending on the nature of each application.
> Also, Spark documentation should describe the conditions that trigger Spark to do this type of serialization asynchronously, so the applications can work around them until the fix is provided. 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org