Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 7 Aug 2014 01:21:11 +0000 (UTC)
From: "Graham Dennis (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12732249.1407313198413.30924.1407374471737@arcas>
In-Reply-To: <JIRA.12732249.1407313198413@arcas>
References: <JIRA.12732249.1407313198413@arcas>
Subject: [jira] [Commented] (SPARK-2878) Inconsistent Kryo serialisation
 with custom Kryo Registrator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SPARK-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088612#comment-14088612 ] 

Graham Dennis commented on SPARK-2878:
--------------------------------------

Stacktrace of the root cause "Failed to run spark.kryo.registrator" error, demonstrating that Kryo deserialisation is happening on a ConnectionManager thread:

14/08/06 18:37:52 ERROR serializer.KryoSerializer: Failed to run spark.kryo.registrator
java.lang.ClassNotFoundException: org.example.MyRegistrator
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:78)
	at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:76)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:76)
	at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:133)
	at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:95)
	at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:995)
	at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1005)
	at org.apache.spark.storage.MemoryStore.getBytes(MemoryStore.scala:102)
	at org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:384)
	at org.apache.spark.storage.BlockManager.getLocalBytes(BlockManager.scala:359)
	at org.apache.spark.storage.BlockManagerWorker.getBlock(BlockManagerWorker.scala:90)
	at org.apache.spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:69)
	at org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44)
	at org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at org.apache.spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:28)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at org.apache.spark.storage.BlockMessageArray.map(BlockMessageArray.scala:28)
	at org.apache.spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:44)
	at org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34)
	at org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34)
	at org.apache.spark.network.ConnectionManager.org$apache$spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:662)
	at org.apache.spark.network.ConnectionManager$$anon$9.run(ConnectionManager.scala:504)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


> Inconsistent Kryo serialisation with custom Kryo Registrator
> ------------------------------------------------------------
>
>                 Key: SPARK-2878
>                 URL: https://issues.apache.org/jira/browse/SPARK-2878
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.0, 1.0.2
>         Environment: Linux RedHat EL 6, 4-node Spark cluster.
>            Reporter: Graham Dennis
>
> The custom Kryo Registrator (a class with the org.apache.spark.serializer.KryoRegistrator trait) is not used with every Kryo instance created, and this causes inconsistent serialisation and deserialisation.
> The Kryo Registrator is sometimes not used because of a ClassNotFound exception that only occurs if it *isn't* the Worker thread (of an Executor) that tries to create the KryoRegistrator.
> A complete description of the problem and a project reproducing the problem can be found at https://github.com/GrahamDennis/spark-kryo-serialisation
> I have currently only tested this with Spark 1.0.0, but will try to test against 1.0.2.


--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org