crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CRUNCH-571) Scrunch functions fail serialization check in the REPL
Date Wed, 14 Oct 2015 13:21:05 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956912#comment-14956912
] 

Tom White edited comment on CRUNCH-571 at 10/14/15 1:20 PM:
------------------------------------------------------------

To reproduce run

{code}
import org.apache.crunch.scrunch.Mem
case class A(x: Int)
val pColOfA = Mem.collectionOf(A(1), A(2))
pColOfA.filter(row => row.x == 1)
{code}

which results in 

{noformat}
java.lang.IllegalStateException: anon$2 named 'null' cannot be serialized
  at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:110)
  at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:129)
  at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:123)
  at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:118)
  at org.apache.crunch.impl.mem.collect.MemCollection.filter(MemCollection.java:275)
  at org.apache.crunch.scrunch.PCollectionLike$class.filter(PCollectionLike.scala:106)
  at org.apache.crunch.scrunch.PCollection.filter(PCollection.scala:29)
  ... 33 elided
Caused by: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException:
$anonfun$1
  at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:165)
  at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:192)
  at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:108)
  ... 39 more
Caused by: java.lang.ClassNotFoundException: $anonfun$1
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:274)
  at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
  at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
  at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
  at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:162)
  ... 41 more
{noformat}


was (Author: tomwhite):
To reproduce run

{code}
import org.apache.crunch.scrunch.Mem
case class A(x: Int)
val pColOfA = Mem.collectionOf(A(1), A(2))
pColOfA.filter(row => row.x == 1)
{code}

which results in 

{noformat}
java.lang.IllegalStateException: anon$2 named 'null' cannot be serialized
  at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:115)
  at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:177)
  at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:171)
  at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:166)
  at org.apache.crunch.impl.mem.collect.MemCollection.filter(MemCollection.java:323)
  at org.apache.crunch.scrunch.PCollectionLike$class.filter(PCollectionLike.scala:106)
  at org.apache.crunch.scrunch.PCollection.filter(PCollection.scala:29)
  ... 33 elided
Caused by: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException:
$anonfun$1
  at org.apache.crunch.impl.mem.collect.MemCollection.deserialize(MemCollection.java:142)
  at org.apache.crunch.impl.mem.collect.MemCollection.deserialize(MemCollection.java:161)
  at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:113)
  ... 39 more
Caused by: java.lang.ClassNotFoundException: $anonfun$1
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:274)
  at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
  at org.apache.crunch.impl.mem.collect.MemCollection$1.resolveClass(MemCollection.java:135)
  at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
  at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
  at org.apache.crunch.impl.mem.collect.MemCollection.deserialize(MemCollection.java:139)
  ... 41 more
{noformat}

> Scrunch functions fail serialization check in the REPL
> ------------------------------------------------------
>
>                 Key: CRUNCH-571
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-571
>             Project: Crunch
>          Issue Type: Bug
>          Components: Scrunch
>    Affects Versions: 0.12.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: CRUNCH-571.patch
>
>
> Calling {{pcol.filter(row => row.x == 1)}} results in {{java.lang.IllegalStateException:
anon$2 named 'null' cannot be serialized}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message