pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode
Date Wed, 20 Sep 2017 04:49:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172741#comment-16172741
] 

liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:48 AM:
----------------------------------------------------------------

{quote}
I also checked, test-tez was not running properly since the Spark 2 support commit, because
setTezEnv was clearing the excluded sources property. I fixed this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in  the dependency of {{test-tez}} in PIG-5157,
But in {{setTezEnv}}, it will reset {{src.exclude.dir}} , this will influence {{jar}} which
use the property {{src.exclude.dir}}?
{code}
 <target name="jar" description="Create pig jar with Spark 1 and 2">
        <echo>Compiling against Spark 2</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="2"/>
        <propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
        <antcall target="jar-core" inheritRefs="true" inheritall="true"/>
        <move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>

        <echo>Compiling against Spark 1</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="1"/>
        <propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
        <antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
        <jar update="yes" jarfile="${output.jarfile.core}">
            <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
        </jar>
        <delete file="${basedir}/_pig-shims.jar"/>
    </target>
{code}

If my understanding is right, why TestEvalPipeline unit test passes in tez mode before this
jira? Seem there are no unit test failures in tez mode before this jira.



was (Author: kellyzly):
{quote}
I also checked, test-tez was not running properly since the Spark 2 support commit, because
setTezEnv was clearing the excluded sources property. I fixed this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in  the dependency of {{test-tez}} in PIG-5157,
But in {{setTezEnv}}, it will reset {{src.exclude.dir}} , this will influence {{jar}} which
use the property {{src.exclude.dir}}?
{code}
 <target name="jar" description="Create pig jar with Spark 1 and 2">
        <echo>Compiling against Spark 2</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="2"/>
        <propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
        <antcall target="jar-core" inheritRefs="true" inheritall="true"/>
        <move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>

        <echo>Compiling against Spark 1</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="1"/>
        <propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
        <antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
        <jar update="yes" jarfile="${output.jarfile.core}">
            <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
        </jar>
        <delete file="${basedir}/_pig-shims.jar"/>
    </target>
{code}

If my understanding is right, why TestEvalPipeline unit test passes in tez mode before this
jira?


> Enable yarn-client mode execution of tests in Spark (1) mode
> ------------------------------------------------------------
>
>                 Key: PIG-5305
>                 URL: https://issues.apache.org/jira/browse/PIG-5305
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>         Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message