hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3565) JavaSerialization can throw java.io.StreamCorruptedException
Date Mon, 16 Jun 2008 11:09:45 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated HADOOP-3565:
------------------------------

    Attachment: hadoop-3565.patch

This happens because java.io.ObjectOutputStream can write back-references to instances previously
serialized in the stream, and after the MapReduce shuffle the references can appear before
the instances they point to under some circumstances.

Here's a patch to fix this, which writes unshared objects, and also resets the stream to clear
shared classes. I've written a unit test too, which fails without this fix. 

> JavaSerialization can throw java.io.StreamCorruptedException
> ------------------------------------------------------------
>
>                 Key: HADOOP-3565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3565
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0, 0.18.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.17.1, 0.18.0
>
>         Attachments: hadoop-3565.patch
>
>
> When using JavaSerialization I sometimes get:
> {noformat}
> Caused by: java.io.StreamCorruptedException: illegal handle value
>         at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1406)
>         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1285)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>         at org.apache.hadoop.io.serializer.JavaSerialization$JavaSerializationDeserializer.deseriali
> ze(JavaSerialization.java:53)
>         at org.apache.hadoop.io.serializer.JavaSerialization$JavaSerializationDeserializer.deseriali
> ze(JavaSerialization.java:36)
>         at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:735)
>         at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:675)
>         ... 4 more
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message