pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (PIG-3978) Container reuse does not across PigServer
Date Mon, 02 Jun 2014 21:15:02 GMT

     [ https://issues.apache.org/jira/browse/PIG-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Daniel Dai resolved PIG-3978.

      Resolution: Fixed
    Hadoop Flags: Reviewed

Patch committed to trunk. Thanks for Rohini for review (review notes is on RB)!

> Container reuse does not across PigServer
> -----------------------------------------
>                 Key: PIG-3978
>                 URL: https://issues.apache.org/jira/browse/PIG-3978
>             Project: Pig
>          Issue Type: Bug
>          Components: tez
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.14.0
>         Attachments: PIG-3978-1.patch
> Container reuse across PigServer does not happen in current code. We use different TezResourceManager
per PigServer, and different TezResourceManager will upload LocalResource to different staging
directory, so LocalResource is different for different PigServer, thus no container reuse
across PigServer. This will affect Pig embedding where multiple PigServer is used for a Pig
> The issue is partly due to PIG-3785 which we switch TezResourceManager to non-singleton.
The root cause for PIG-3785 is not the singleton issue. The real issue is TezResourceManager.addTezResource
is not synchronized, there is a race condition when one PigServer already upload the resource
and use it in container, and the other PigServer overwrite at the same time.  

This message was sent by Atlassian JIRA

View raw message