crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CRUNCH-511) Scrunch product type support should use derived() instead of derivedImmutable()
Date Sat, 18 Apr 2015 15:23:58 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Wills resolved CRUNCH-511.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 0.12.0

Pushed to master.

> Scrunch product type support should use derived() instead of derivedImmutable()
> -------------------------------------------------------------------------------
>
>                 Key: CRUNCH-511
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-511
>             Project: Crunch
>          Issue Type: Bug
>          Components: Scrunch
>            Reporter: Josh Wills
>             Fix For: 0.12.0
>
>         Attachments: CRUNCH-511.patch, CRUNCH-511b.patch
>
>
> Ran into a subtle bug related to Scrunch case class support: right now, I was using derivedImmutable
to generate the mapping from an Avro record in to a Scala case class, thinking (foolishly,
as it turned out) that case classes were always immutable. Although that's true, the arguments
that get passed to the case class constructor (the fields of the Avro record) are not necessarily
immutable (e.g., an Avro ByteBuffer gets reused from key to key in a reduce), and those values
can cause join key errors.
> I wrote a test case to illustrate the bug and switched the case class PType generator
to use derived instead of derivedImmutable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message