pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4846) Use pigmix to test the performance of pig on spark
Date Mon, 18 Apr 2016 07:58:25 GMT

    [ https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245259#comment-15245259

liyunzhang_intel commented on PIG-4846:

[~xuefuz]:  Thanks for your configuration.
spark.yarn.driver.memoryOverhead is a parameter for yarn-cluster mode, currently, we don't
support yarn-cluster mode for pig on spark(see PIG-4681), so i don't add this parameter in
the pig.properties.

I add following in conf/pig.properties
Here the unit of spark.yarn.executor.memoryOverhead is MB.

The new result of yarn-client mode is 

It shows a big peformance improvement than before :).
Can you explain more about how to configure these configuration?

> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>                 Key: PIG-4846
>                 URL: https://issues.apache.org/jira/browse/PIG-4846
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>             Fix For: spark-branch
>         Attachments: PIG-4846.patch, PIG-4846_1.patch
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.

This message was sent by Atlassian JIRA

View raw message