crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-300) Support reflected Avro record writing from MemPipeline
Date Thu, 21 Nov 2013 20:21:36 GMT


Gabriel Reid updated CRUNCH-300:

    Attachment: CRUNCH-300b.patch

Patch made compatible with CRUNCH-293, and added some more integration tests for the changed

This patch also makes the writing and filenaming of output files from MemPipeline consistent
-- the given output path is taken as a directory, and a new file is created in the directory.
This is the way that Avro and text files already worked, but it is a change for how the sequence
file writing worked.

> Support reflected Avro record writing from MemPipeline
> ------------------------------------------------------
>                 Key: CRUNCH-300
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: David Whiting
>            Assignee: Gabriel Reid
>            Priority: Minor
>         Attachments: 0001-Allow-MemPipeline-to-write-Avro-files-by-reflection.patch,
> MemPipeline doesn't support writing Avro records via reflection. It seems that this was
half implemented but never finished, but I needed it to create some test data to run through
a cluster MapReduce test. The current implementation correctly reflects the schema, but then
uses a GenericDatumWriter to try and write the record, causing a ClassCastException. The correct
way would be to get a ReflectDatumWriter from the ReflectDataFactory.

This message was sent by Atlassian JIRA

View raw message