incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-71) PType mapping functions are not initialized before being used for deep copying
Date Thu, 20 Sep 2012 12:38:07 GMT


Gabriel Reid updated CRUNCH-71:

    Attachment: CRUNCH-71.patch

Patch to resolve the issue attached.

I'm not 100% happy with this solution, as I would prefer that the PType would be supplied
to the DoFn at runtime instead of the DoFn being responsible for calling PType#initialize.

However, that approach could bring a lot of extra work along with it in job setup as there
is not a 1-to-1 relationship between DoFns and PTypes. 

As object reuse issues are pretty isolated in MR contexts (joins are the main place where
I see them occurring) then this fix feels ok to me for now. Any objections to this patch?
> PType mapping functions are not initialized before being used for deep copying
> ------------------------------------------------------------------------------
>                 Key: CRUNCH-71
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>    Affects Versions: 0.3.0
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>         Attachments: CRUNCH-71.patch
> The PType#getDetachedValue method performs a deep copy (if needed) in order to allow
DoFns to hold on to values that have been passed through them (for example, in join functions).
> The WritablePType class uses the built-in input and output MapFns in the PType to handle
this deep copying, but the input and output MapFns don't get initialized (i.e. initialize
isn't called on them) after they are deserialized along with the DoFn that is using them.
In some rare cases (at least for tuples), this can result in NullPointerExceptions or other

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message