crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-556) Fix total sorts in Crunch-on-Spark
Date Thu, 13 Aug 2015 18:44:46 GMT


Josh Wills commented on CRUNCH-556:

Yeah, so I think to [~smungre]'s point, we need to add integration tests between HBase and
Spark if we expect that Crunch-on-Spark will play nice w/HBase. There's also no way to really
test the total sort patch w/o starting up a mini cluster (I think, could be wrong.) And so
I'm procrastinating doing that by pretending to think really hard about the right way to do
it-- new module? crunch-spark as a test dependency for crunch-hbase? Or crunch-hbase as a
test dependency for crunch-spark?

> Fix total sorts in Crunch-on-Spark
> ----------------------------------
>                 Key: CRUNCH-556
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 0.13.0
>            Reporter: Josh Wills
>             Fix For: 0.14.0
>         Attachments: CRUNCH-556.patch
> From the user mailing list, trying to perform a total sort to create an HFile w/Crunch
on Spark throws the following exception:
> The problem can be traced to not properly configuring the partitioner w/the path to the
partition file that is stored in the GroupingOptions extra configuration settings. These settings
get passed correctly for the MR job, but not for the Spark ones.

This message was sent by Atlassian JIRA

View raw message