pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-1815) pig task retains used instances of PhysicalPlan
Date Thu, 20 Jan 2011 23:54:45 GMT
pig task retains used instances of PhysicalPlan
-----------------------------------------------

                 Key: PIG-1815
                 URL: https://issues.apache.org/jira/browse/PIG-1815
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.8.0
            Reporter: Thejas M Nair
            Assignee: Thejas M Nair
             Fix For: 0.9.0, 0.8.0


map tasks of a pig query ran out of memory because there were too many (thousands)  instances
of combiner PhysicalPlan in memory. Each physical plan (except the last?) was linked to older
one as shown in the yourkit snapshot that I am attaching.

This problem was noticed with 0.8 because of the split combination feature, that resulted
in each map having larger inputs. The query also had large physical plan because of multi-query,
it had 17 MR jobs merged into one during the multi-query optimization phase.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message