pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohit Sabharwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4374) Add SparkPlan in spark package
Date Tue, 27 Jan 2015 01:06:34 GMT

    [ https://issues.apache.org/jira/browse/PIG-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292773#comment-14292773
] 

Mohit Sabharwal commented on PIG-4374:
--------------------------------------

[~kellyzly], I'm not sure we need these classes if the purpose is to fix the tests.  These
look like versions of MROperPlan, MRCompiler, etc, without MR specific attributes. 

I think unit tests are failing because JobGraph is not correctly populated in spark's implementation
of PigStats. We can correctly populate the JobGraph without adding SparkCompiler, SparkOperPlan,
etc.  JobGraph needs as many JobStats as there are Spark jobs in the pig script.  For example:
see PIG-4393 for the approach.

What do you think ?



> Add SparkPlan in spark package
> ------------------------------
>
>                 Key: PIG-4374
>                 URL: https://issues.apache.org/jira/browse/PIG-4374
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-4374_1.patch, Pig-spark #76 [Jenkins .png, jenkins_PIG-4374_1_patch.png.png
>
>
> in current code, there are following class in mapreduce and tez package.
> mapreduce:
> MRCompiler
> MROperPlan  MROpPlanVisitor
> MapReduceOper
> tez:
> TezCompiler
> TezOperPlan  TezOpPlanVisitor
> TezOperator
> Following class needs to be added in spark package:
> SparkCompiler
> SparkOperPlan  SparkOpPlanVisitor
> SparkOperator
> Current code need to be refactored because of above adding classes. Some unit tests like
TestStoreInstances fail because of it.
> following are the error of unit test TestStoreInstances:
> Error Message
> num jobs expected:<1> but was:<0>
> Stacktrace
> junit.framework.AssertionFailedError: num jobs expected:<1> but was:<0>
> 	at org.apache.pig.test.TestStoreInstances.testBackendStoreCommunication(TestStoreInstances.java:122)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message