crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-556) Fix total sorts in Crunch-on-Spark
Date Thu, 13 Aug 2015 18:44:46 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695746#comment-14695746
] 

Josh Wills commented on CRUNCH-556:
-----------------------------------

Yeah, so I think to [~smungre]'s point, we need to add integration tests between HBase and
Spark if we expect that Crunch-on-Spark will play nice w/HBase. There's also no way to really
test the total sort patch w/o starting up a mini cluster (I think, could be wrong.) And so
I'm procrastinating doing that by pretending to think really hard about the right way to do
it-- new module? crunch-spark as a test dependency for crunch-hbase? Or crunch-hbase as a
test dependency for crunch-spark?

> Fix total sorts in Crunch-on-Spark
> ----------------------------------
>
>                 Key: CRUNCH-556
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-556
>             Project: Crunch
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 0.13.0
>            Reporter: Josh Wills
>             Fix For: 0.14.0
>
>         Attachments: CRUNCH-556.patch
>
>
> From the user mailing list, trying to perform a total sort to create an HFile w/Crunch
on Spark throws the following exception: https://gist.github.com/anonymous/920c000f20229eaa76d8
> The problem can be traced to not properly configuring the partitioner w/the path to the
partition file that is stored in the GroupingOptions extra configuration settings. These settings
get passed correctly for the MR job, but not for the Spark ones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message