Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Date: Thu, 24 Mar 2016 17:37:25 +0000 (UTC)
From: "Josh Wills (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12952456.1458657800000.43114.1458841045493@Atlassian.JIRA>
In-Reply-To: <JIRA.12952456.1458657800000@Atlassian.JIRA>
References: <JIRA.12952456.1458657800000@Atlassian.JIRA>
 <JIRA.12952456.1458657800794@arcas>
Subject: [jira] [Commented] (CRUNCH-598) scaleFactor for JoinStrategy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210610#comment-15210610 ] 

Josh Wills commented on CRUNCH-598:
-----------------------------------

[~desmit] what do you imagine as a fix here? A constructor argument for DefaultJoinStrategy (and possibly ShardedJoinStrategy?)

> scaleFactor for JoinStrategy
> ----------------------------
>
>                 Key: CRUNCH-598
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-598
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Stefan De Smit
>            Priority: Minor
>
> the scaleFactor method has a big influence on planner.
> For joins, there currently isn't a clean way to set this, while it often is required, as a join can have a big multiply factor.
> for the DefaultJoinStrategy, it's possible to add a custom JoinFn with proper scaleFactor, or just extend the default InnerJoinFn with a scaleFactor.
> For the ShardedJoinStrategy, this isn't possible, while it often is needed more (as ShardedJoin is especially handy for 1 to really many).
> For the default ConstantShardingStrategy, it might make sense to use the numShards also as scalingFactor for left side. as that's kind of what happens: emit every left entry numShards times.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)