pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4846) Use pigmix to test the performance of pig on spark
Date Wed, 13 Apr 2016 13:40:25 GMT

    [ https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239246#comment-15239246
] 

Xuefu Zhang commented on PIG-4846:
----------------------------------

Your machine seems having abundant cores but scarce memory. I suggest the following:

YARN configuration:
{code}
yarn.nodemanager.resource.memory-mb=56G
 yarn.nodemanger.resource.cpu-vcores=28
{code}

Spark configurations:
{code}
spark.executor.cores=4
spark.executor.memory=6.4G
spark.yarn.executor.memoryOverhead=1.6G
spark.driver.memory=2G
spark.yarn.driver.memoryOverhead=400M
spark.executor.instances=7
{code}
Please note that the numbers might need to be converted to the unit of individual property.
For instance, .memory takes bytes while memoryOverhead takes MB.



> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
>                 Key: PIG-4846
>                 URL: https://issues.apache.org/jira/browse/PIG-4846
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>             Fix For: spark-branch
>
>         Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message