crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-23) PCollection#sort doesn't do a full sort on values
Date Fri, 17 Aug 2012 03:56:38 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436546#comment-13436546
] 

Rahul Sharma commented on CRUNCH-23:
------------------------------------

Yes the patch is the full cumulative patch.  It works on SequenceFiles only. Basically in
the end the TotalOrderPartioner requires a SequenceFile with sorted keys and null values.
We need keys for this file, for which we need to materialize an intermediate collection, and
I think when we do so then crunch creates intermediate output files which are SequenceFiles
only. I am not completely sure on this but till now I haven't seen a case where it creates
a different type out intermediate output file. Also I haven't tested it with Avro files.
                
> PCollection#sort doesn't do a full sort on values
> -------------------------------------------------
>
>                 Key: CRUNCH-23
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-23
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Gabriel Reid
>            Assignee: Rahul Sharma
>         Attachments: 0001-CRUNCH-23-fix-sorting.patch, CRUNCH-23-sorting-issue.patch,
CRUNCH-23-used-TotalOrderpartioner-for-sorting-keys.patch, SortTest.java
>
>
> When a PCollection is sorted (using PCollection#sort), the sorting that is performed
is only per reducer, and not an absolute sort over all values. This means that the values
are not in sorted order if they are iterated over on a materialized collection. It also means
that the sorted files that are output from a sort operation can not be simply concatenated
to come to a single sorted file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message