crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-448) Smart-ish case class serialization for Scrunch
Date Mon, 21 Jul 2014 16:44:39 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Wills updated CRUNCH-448:
------------------------------

    Attachment: CRUNCH-448.patch

The patch, with tests. I have a couple more revisions I'd like to do on this one, but this
is the basics.

> Smart-ish case class serialization for Scrunch
> ----------------------------------------------
>
>                 Key: CRUNCH-448
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-448
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Scrunch
>            Reporter: Josh Wills
>         Attachments: CRUNCH-448.patch
>
>
> I took advantage of some of the new reflection features in Scala 2.10 to make automatic
serialization of case classes much cooler, viz.,
> 1) We can automatically map any case class to a TupleN and properly infer PTypes for
Scala types (options, lists, etc.), just as we do for regular PCollections.
> 2) For Avro serialization, we will also capture the names of the case class' fields and
map them to the names of the fields in the Avro record we generate, which makes debugging
intermediate outputs much easier.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message