pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (PIG-4703) TezOperator.stores shall not ship to backend
Date Thu, 15 Oct 2015 17:56:05 GMT
Daniel Dai created PIG-4703:

             Summary: TezOperator.stores shall not ship to backend
                 Key: PIG-4703
                 URL: https://issues.apache.org/jira/browse/PIG-4703
             Project: Pig
          Issue Type: Bug
          Components: tez
            Reporter: Daniel Dai
            Assignee: Daniel Dai
             Fix For: 0.16.0, 0.15.1
         Attachments: PIG-4703-1.patch

We end up of shipping all physical plan to the backend. One direct result is UDFContext is
not set in OutputCommitter. Here is how:
1. PigGraceShuffleVertexManager deserialize "pig.tez.plan". TezOperator.stores reference physical
plan so physical plan get deserialize.
2. When deserialize POUserFunc, POUserFunc.instantiateFunc is invoked. Inside it, POUserFunc.setFuncInputSchema
would use UDFContext and leave an entry there (though it is empty map).
3. MapRedUtil.setupUDFContext would only deserialize if UDFContext is empty, since we already
touch UDFContext with some maps, UDFContext is not deserialized

This message was sent by Atlassian JIRA

View raw message