pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Ding (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4173) Move to Spark 1.x
Date Fri, 03 Oct 2014 20:56:34 GMT

     [ https://issues.apache.org/jira/browse/PIG-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Richard Ding updated PIG-4173:
------------------------------
    Attachment: PIG-4174_4.patch

This patch fixed the unit tests. 

The version of Spark used is 1.0.2. In Spark 1.1.0, the CoGroupRDD is changed and breaks the
cogroup runtime. I'm looking into this.

> Move to Spark 1.x
> -----------------
>
>                 Key: PIG-4173
>                 URL: https://issues.apache.org/jira/browse/PIG-4173
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: bc Wong
>            Assignee: Richard Ding
>         Attachments: PIG-4173.patch, PIG-4173_2.patch, PIG-4173_3.patch, PIG-4174_4.patch,
TEST-org.apache.pig.spark.TestSpark.txt
>
>
> The Spark branch is using Spark 0.9: https://github.com/apache/pig/blob/spark/ivy.xml#L438.
We should probably switch to Spark 1.x asap, due to Spark interface changes since 1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message