Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B79AD10953 for ; Fri, 14 Feb 2014 20:52:26 +0000 (UTC) Received: (qmail 22765 invoked by uid 500); 14 Feb 2014 20:52:25 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 22737 invoked by uid 500); 14 Feb 2014 20:52:24 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 22729 invoked by uid 500); 14 Feb 2014 20:52:23 -0000 Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org Received: (qmail 22561 invoked by uid 99); 14 Feb 2014 20:52:21 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Feb 2014 20:52:21 +0000 Date: Fri, 14 Feb 2014 20:52:21 +0000 (UTC) From: "Gabriel Reid (JIRA)" To: crunch-dev@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CRUNCH-329) Re-add type info to TupleWritable to make fields sort correctly MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CRUNCH-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901921#comment-13901921 ] Gabriel Reid commented on CRUNCH-329: ------------------------------------- +1, looks good to me, and I like how you got the reading of the configured serialization codes in there. > Re-add type info to TupleWritable to make fields sort correctly > --------------------------------------------------------------- > > Key: CRUNCH-329 > URL: https://issues.apache.org/jira/browse/CRUNCH-329 > Project: Crunch > Issue Type: Bug > Components: Core > Affects Versions: 0.10.0, 0.8.3 > Reporter: Josh Wills > Assignee: Josh Wills > Fix For: 0.10.0, 0.8.3 > > Attachments: CRUNCH-329.patch, CRUNCH-329b.patch, fix-ss-writables.patch > > > Secondary sorts aren't currently working correctly for Writable types after we hacked the TupleWritable impl to make all of the fields BytesWritables (e.g., secondary IntWritable values will no longer be sorted correctly, even though everything is still grouped correctly.) > The least-bad way that I came up with to fix this is to use integer codes for each possible WritableComparable type in a pipeline that we can use to decode what Writable type each tuple field corresponds to. This allows us to keep the various fields sortable while still doing a reasonable job of minimizing the serialization required to pass the type information along. -- This message was sent by Atlassian JIRA (v6.1.5#6160)