crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-571) Scrunch functions fail serialization check in the REPL
Date Wed, 14 Oct 2015 13:19:05 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated CRUNCH-571:
-----------------------------
    Attachment: CRUNCH-571.patch

The problem is that {{ObjectInputStream}} (which is used by {{SerializationUtils}} from Apache
Commons) does some magic tricks to find a classloader, but these don't work in the REPL context.
The problem is explained well in https://issues.scala-lang.org/browse/SI-2403, however the
fix there doesn't quite work in our case, since the Scrunch classes are loaded by the application
class loader, not the REPL classloader, so the class for the anonymous function can still
not be found.

The REPL does set the thread context classloader, however, so a slight adaptation of the approach
in SI-2403 does work if we try the thread context classloader, which is what this patch does.

There's no unit test as this problem only manifests itself in the REPL, however I have manually
tested the fix with success.

> Scrunch functions fail serialization check in the REPL
> ------------------------------------------------------
>
>                 Key: CRUNCH-571
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-571
>             Project: Crunch
>          Issue Type: Bug
>          Components: Scrunch
>    Affects Versions: 0.12.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: CRUNCH-571.patch
>
>
> Calling {{pcol.filter(row => row.x == 1)}} results in {{java.lang.IllegalStateException:
anon$2 named 'null' cannot be serialized}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message