crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan De Smit (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-598) scaleFactor for JoinStrategy
Date Tue, 22 Mar 2016 14:43:25 GMT
Stefan De Smit created CRUNCH-598:
-------------------------------------

             Summary: scaleFactor for JoinStrategy
                 Key: CRUNCH-598
                 URL: https://issues.apache.org/jira/browse/CRUNCH-598
             Project: Crunch
          Issue Type: Improvement
            Reporter: Stefan De Smit
            Priority: Minor


the scaleFactor method has a big influence on planner.
For joins, there currently isn't a clean way to set this, while it often is required, as a
join can have a big multiply factor.
for the DefaultJoinStrategy, it's possible to add a custom JoinFn with proper scaleFactor,
or just extend the default InnerJoinFn with a scaleFactor.
For the ShardedJoinStrategy, this isn't possible, while it often is needed more (as ShardedJoin
is especially handy for 1 to really many).
For the default ConstantShardingStrategy, it might make sense to use the numShards also as
scalingFactor for left side. as that's kind of what happens: emit every left entry numShards
times.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message