crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-329) Re-add type info to TupleWritable to make fields sort correctly
Date Thu, 23 Jan 2014 04:04:33 GMT
Josh Wills created CRUNCH-329:
---------------------------------

             Summary: Re-add type info to TupleWritable to make fields sort correctly
                 Key: CRUNCH-329
                 URL: https://issues.apache.org/jira/browse/CRUNCH-329
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.10.0, 0.8.3
            Reporter: Josh Wills
            Assignee: Josh Wills
             Fix For: 0.10.0, 0.8.3


Secondary sorts aren't currently working correctly for Writable types after we hacked the
TupleWritable impl to make all of the fields BytesWritables (e.g., secondary IntWritable values
will no longer be sorted correctly, even though everything is still grouped correctly.)

The least-bad way that I came up with to fix this is to use integer codes for each possible
WritableComparable type in a pipeline that we can use to decode what Writable type each tuple
field corresponds to. This allows us to keep the various fields sortable while still doing
a reasonable job of minimizing the serialization required to pass the type information along.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message