pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2
Date Fri, 01 Dec 2017 18:50:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274766#comment-16274766

Rohini Palaniswamy commented on PIG-5318:

You should just do isSpark2_x (sparkVersion.startsWith("2.")) instead of isSpark2_2_x . If
Spark 2.3 gets released, then code will have to change. 

bq. Not sure if it is a bug in Pig, or in Spark, should Spark consistently use the same OutputFormat
instance in this case?
  Spark should consistently use the same OutputFormat instance in this case. We should not
be modifying the test case. There will be users who will be using local variables in StoreFunc
for some computation at least.

> Unit test failures on Pig on Spark with Spark 2.2
> -------------------------------------------------
>                 Key: PIG-5318
>                 URL: https://issues.apache.org/jira/browse/PIG-5318
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>         Attachments: PIG-5318_1.patch, PIG-5318_2.patch
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed by asserting
on the message of the exception's root cause, looks like on Spark 2.2 the exception is wrapped
into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks like SPARK-7953
is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.

This message was sent by Atlassian JIRA

View raw message