pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2
Date Mon, 04 Dec 2017 17:26:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277120#comment-16277120
] 

Rohini Palaniswamy commented on PIG-5318:
-----------------------------------------

bq. but how about modifying PigOutputformat, like I did in the patch (making the relevant
variables static)?
 This cannot be done. It is hacky and will break Pig local mode and Tez. In local mode, save
jvm is used to execute the whole script which can have parallel STORE statements. Tez also
allows storing to multiple outputs from same vertex in a DAG - i.e multiple PigOutputFormat
in the save jvm.

bq. isSpark2_1_minus
  Can you make it  isSpark2_2_plus which is slightly more intuitive than 2_1_minus. Also instantiating
SparkContext just to get version seems overkill. Prefer the previous logic you had. Is there
any reason that could not be used?


> Unit test failures on Pig on Spark with Spark 2.2
> -------------------------------------------------
>
>                 Key: PIG-5318
>                 URL: https://issues.apache.org/jira/browse/PIG-5318
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>         Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, PIG-5318_4.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed by asserting
on the message of the exception's root cause, looks like on Spark 2.2 the exception is wrapped
into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks like SPARK-7953
is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message