pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4523) Tez engine should use tez config rather than mr config whenever possible
Date Wed, 13 May 2015 23:34:59 GMT

     [ https://issues.apache.org/jira/browse/PIG-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Daniel Dai updated PIG-4523:
      Resolution: Fixed
    Hadoop Flags: Reviewed
          Status: Resolved  (was: Patch Available)

Patch committed to branch 0.15 and trunk. Thanks Rohini for review!

> Tez engine should use tez config rather than mr config whenever possible
> ------------------------------------------------------------------------
>                 Key: PIG-4523
>                 URL: https://issues.apache.org/jira/browse/PIG-4523
>             Project: Pig
>          Issue Type: Bug
>          Components: tez
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.15.0
>         Attachments: PIG-4523-1.patch, PIG-4523-2.patch
> Currently, Pig on tez use memory/heap setting from MR not tez, but respect other tez
settings. In the environment which MR and tez settings are not aligned, this could be a problem.
For a specific scenario, user increase both tez both memory settings and io.sort.mb to tune
tez performance, but MR settings retains. Pig on Tez get increased io.sort.mb but same vertex
memory, end up with OOM exception.
> I think tez engine shall respect Tez settings first, unless user really want to use mr
settings for backward compatibility. It can be:
> 1. a flag which turn on the mr settings for tez
> 2. If the mr memory settings are from Pig script not the config file

This message was sent by Atlassian JIRA

View raw message