Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Date: Wed, 12 Apr 2017 18:52:41 +0000 (UTC)
From: "Josh Wills (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.13062890.1491829808000.268648.1492023161711@Atlassian.JIRA>
In-Reply-To: <JIRA.13062890.1491829808000@Atlassian.JIRA>
References: <JIRA.13062890.1491829808000@Atlassian.JIRA> <JIRA.13062890.1491829808236@jira-lw-us.apache.org>
Subject: [jira] [Commented] (CRUNCH-642) Enable numReducers option for
 methods in Distinct
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 12 Apr 2017 18:52:46 -0000


    [ https://issues.apache.org/jira/browse/CRUNCH-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966391#comment-15966391 ] 

Josh Wills commented on CRUNCH-642:
-----------------------------------

[~xaviert] it is more flexible, but it involves a backwards incompatible change to the API, which I generally try to avoid; no problem with adding another method that exposes the GroupingOptions, but please do not remove the existing API method here.

> Enable numReducers option for methods in Distinct
> -------------------------------------------------
>
>                 Key: CRUNCH-642
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-642
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.14.0
>            Reporter: Xavier
>            Assignee: Josh Wills
>            Priority: Trivial
>         Attachments: CRUNCH-642-Enable-GroupingOptions-for-Distinct-operations.patch, CRUNCH-642.patch
>
>
> The {{groupByKey}} invocation in the {{Distinct}} class currently uses the default  (recommended) number of reducers without providing an option to override this:
> {code}
> public static <S> PCollection<S> distinct(PCollection<S> input, int flushEvery) {
>   Preconditions.checkArgument(flushEvery > 0);
>   PType<S> pt = input.getPType();
>   PTypeFamily ptf = pt.getFamily();
>   return input
>       .parallelDo("pre-distinct", new PreDistinctFn<S>(flushEvery, pt), ptf.tableOf(pt, ptf.nulls()))
>       .groupByKey()
>       .parallelDo("post-distinct", new PostDistinctFn<S>(), pt);
> }
> {code}
> Would it be possible to enhance this method such that it is possible to customize the number of reducers? Either explicitly or via a {{GroupingOptions}} object.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)