Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Date: Fri, 14 Sep 2012 18:14:12 +1100 (NCT)
From: "Gabriel Reid (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <1557043548.79286.1347606852318.JavaMail.jiratomcat@arcas>
In-Reply-To: <279054043.41247.1346878871463.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (CRUNCH-57) Add a length function to PCollection
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455630#comment-13455630 ] 

Gabriel Reid commented on CRUNCH-57:
------------------------------------

@Rahul, about the MinMaxFn.patch -- I think that this will cause problems if a non-writable class (for example, with a custom compare implementation) is used with a different backing writable class.

Additionally, it won't give a very clear error when the value really isn't comparable, like in the case of a custom writable class that isn't comparable, or a avro-reflection class.

In general I think we want to stay away from basing things on the underlying serialization scheme, as the underlying serialization may have different compare logic (either implicitly or explicitly) than the type of object in the PCollection
                
> Add a length function to PCollection
> ------------------------------------
>
>                 Key: CRUNCH-57
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-57
>             Project: Crunch
>          Issue Type: New Feature
>          Components: Core
>    Affects Versions: 0.3.0
>            Reporter: Kiyan Ahmadizadeh
>            Assignee: Josh Wills
>         Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>  
> For example, suppose there was an initial PCollection that was then filtered into another.  If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira