Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C60FC200D60 for ; Fri, 1 Dec 2017 19:50:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C4B24160C06; Fri, 1 Dec 2017 18:50:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 170D1160C18 for ; Fri, 1 Dec 2017 19:50:03 +0100 (CET) Received: (qmail 2060 invoked by uid 500); 1 Dec 2017 18:50:03 -0000 Mailing-List: contact dev-help@pig.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@pig.apache.org Delivered-To: mailing list dev@pig.apache.org Received: (qmail 2049 invoked by uid 500); 1 Dec 2017 18:50:03 -0000 Delivered-To: apmail-hadoop-pig-dev@hadoop.apache.org Received: (qmail 2046 invoked by uid 99); 1 Dec 2017 18:50:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Dec 2017 18:50:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 43FCD1806C0 for ; Fri, 1 Dec 2017 18:50:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -98.702 X-Spam-Level: X-Spam-Status: No, score=-98.702 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_NUMSUBJECT=0.5, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id BYB8lddy9V_N for ; Fri, 1 Dec 2017 18:50:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 2DF725F46D for ; Fri, 1 Dec 2017 18:50:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id C6536E0E4D for ; Fri, 1 Dec 2017 18:50:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 6B612255C2 for ; Fri, 1 Dec 2017 18:50:00 +0000 (UTC) Date: Fri, 1 Dec 2017 18:50:00 +0000 (UTC) From: "Rohini Palaniswamy (JIRA)" To: pig-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 01 Dec 2017 18:50:05 -0000 [ https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274766#comment-16274766 ] Rohini Palaniswamy commented on PIG-5318: ----------------------------------------- You should just do isSpark2_x (sparkVersion.startsWith("2.")) instead of isSpark2_2_x . If Spark 2.3 gets released, then code will have to change. bq. Not sure if it is a bug in Pig, or in Spark, should Spark consistently use the same OutputFormat instance in this case? Spark should consistently use the same OutputFormat instance in this case. We should not be modifying the test case. There will be users who will be using local variables in StoreFunc for some computation at least. > Unit test failures on Pig on Spark with Spark 2.2 > ------------------------------------------------- > > Key: PIG-5318 > URL: https://issues.apache.org/jira/browse/PIG-5318 > Project: Pig > Issue Type: Bug > Components: spark > Reporter: Nandor Kollar > Assignee: Nandor Kollar > Attachments: PIG-5318_1.patch, PIG-5318_2.patch > > > There are several failing cases when executing the unit tests with Spark 2.2: > {code} > org.apache.pig.test.TestAssert#testNegativeWithoutFetch > org.apache.pig.test.TestAssert#testNegative > org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch > org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput > org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore > org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication > org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore > {code} > All of these are related to fixes/changes in Spark. > TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed by asserting on the message of the exception's root cause, looks like on Spark 2.2 the exception is wrapped into an additional layer. > TestStore and TestStoreLocal failure are also a test related problems: looks like SPARK-7953 is fixed in Spark 2.2 > The root cause of TestStoreInstances is yet to be found out. -- This message was sent by Atlassian JIRA (v6.4.14#64029)