pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4245) 1-1 edge vertices should use same jvm opts
Date Tue, 28 Oct 2014 23:13:33 GMT

    [ https://issues.apache.org/jira/browse/PIG-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187678#comment-14187678

Daniel Dai commented on PIG-4245:


> 1-1 edge  vertices should use same jvm opts
> -------------------------------------------
>                 Key: PIG-4245
>                 URL: https://issues.apache.org/jira/browse/PIG-4245
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.14.0
>         Attachments: PIG-4245-1.patch
>    We use map jvm opts for root vertices and reduce jvm opts for intermediate and leaf
vertices. For a simple order by or skewed join the Sampler will be a root vertex and Partitioner
will be a intermediate vertex. If the map opts and reduce opts are different containers will
not be reused and there will be lot of resource wastage when the reduce opts specify higher
memory than map which is usually the case. The partitioner does not do much and just does
identity mapping. So we should be good using map opts there. 

This message was sent by Atlassian JIRA

View raw message