crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-539) Use of TupleWritable.setConf fails in mapper/reducer
Date Wed, 08 Jul 2015 05:33:04 GMT


Gabriel Reid updated CRUNCH-539:
    Attachment: CRUNCH-539.patch

Patch to resolve this. Instead of using ObjectOutputStream and ObjectInputStream to serialize/deserialize
the BiMap of codes to WritableComparable classes, the map is serialized as simple text (which
also makes it easier to debug).

Tested this in the local job tracker in Hadoop 2.7.0 (also with custom WritableComparables
registered) in code that is pretty much the same as that in the original mail thread (see It isn't possible (or it's at least non-trivial) to add a real of
this in the Crunch integration tests, due to it depending on loading classes that aren't available
to the root class loader.

> Use of TupleWritable.setConf fails in mapper/reducer
> ----------------------------------------------------
>                 Key: CRUNCH-539
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: Gabriel Reid
>         Attachments: CRUNCH-539.patch
> In (at least) more recent versions of Hadoop 2, the implicit call to TupleWritable.setConf
that happens when using TupleWritables fails with a ClassNotFoundException for (ironically)
the TupleWritable class.
> This appears to be due to the way that ObjectInputStream resolves classes in its [resolveClass
together with the way that the context classloader is set within a hadoop mapper or reducer.
> This is similar to PIG-2532.
> This can be reproduced in the local job tracker (at least) in Hadoop 2.7.0, but it can't
be reproduced in Crunch integration tests (due to classloading setup). It appears that this
issue is only present in Crunch 0.12.
> The following code within a simple pipeline will cause this issue to occur:
> {code}
> PTable<String, Integer> yearTemperatures = ... /* Writable-based PTable */
> PTable<String, Integer> maxTemps = yearTemperatures
>                 .groupByKey()
>                 .combineValues(Aggregators.MAX_INTS())
>                 .top(1);   //LINE THAT CAUSES THE ERROR
> {code}

This message was sent by Atlassian JIRA

View raw message