pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nandor Kollar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2
Date Mon, 11 Dec 2017 09:01:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285663#comment-16285663

Nandor Kollar commented on PIG-5318:

[~rohini] indeed, it is internal, but I think checking for existence of {{spark-version-info.properties}}
and the property value inside it is also Spark's internal. The public API is {{SparkContext#version()}},
but that means we should create a SparkContext just for telling the version, and we agreed
that it is overkill.

As for SPARK-7953, I think SPARK-18219 fixed this problem too. It looks like in Spark 2.2
they completely refactored the task and job commit protocol: they [moved|https://github.com/apache/spark/commit/937af592e65f4dd878aafcabf8fe2cfe7fa3d9b3#diff-d97cfb5711116287a7655f32cd5675cb]
the Spark SQL implementation to the Spark code module. I asked on the mentioned Jira if it
is still an outstanding problem or not (maybe I missed some detail), I'd like to give some
time for answers before I close it.

> Unit test failures on Pig on Spark with Spark 2.2
> -------------------------------------------------
>                 Key: PIG-5318
>                 URL: https://issues.apache.org/jira/browse/PIG-5318
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>         Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, PIG-5318_4.patch,
PIG-5318_5.patch, PIG-5318_6.patch
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed by asserting
on the message of the exception's root cause, looks like on Spark 2.2 the exception is wrapped
into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks like SPARK-7953
is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.

This message was sent by Atlassian JIRA

View raw message