pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Graham (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1313) PigServer leaks memory over time
Date Thu, 01 Apr 2010 16:46:27 GMT

     [ https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Bill Graham updated PIG-1313:

    Attachment: PIG-1313-4.patch

# Attaching PIG-1313-4.patch with additional Javadocs on PigServer and PigServer.shutdown().

# triggerDeleteOnFail is still called by TestMultQueryLocal.executePlan, but you are correct
in that no one is (or was) calling registerDeleteOnFail, which is the only entry point to
push something onto the deleteOnFail stack.

I will gladly remove deleteOnFail and all calls to it as part of this JIRA, or we can handle
it in another one if that's cleaner w.r.t. issue tracking. Let me know. 

> PigServer leaks memory over time
> --------------------------------
>                 Key: PIG-1313
>                 URL: https://issues.apache.org/jira/browse/PIG-1313
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Bill Graham
>            Assignee: Bill Graham
>         Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch, PIG-1313-1.patch, PIG-1313-2.patch,
PIG-1313-3.patch, PIG-1313-4.patch, Pig1313Reproducer.java
> When {{PigServer}} runs it creates temporary files using the {{FileLocalizer.getTemporaryPath(..)}}.
This static method creates and returns a handle to a temporary file (as an instance of {{ElementDescriptor}}).
The {{ElementDescriptors}} returned by this method are kept on a static {{Stack}} named {{toDelete}}.
The items on {{toDelete}} get removed by the {{FileLocalizer.deleteTempFile()}} method.
> The only place in the code where I see {{FileLocalizer.deleteTempFile()}} called is in
the Main class. {{PigServer}} does not call that method though, so a long-running VM that
repeatedly uses instances of {{PigServer}} to run jobs will leak memory via {{toDelete}}.
> One suggested fix is to have {{PigServer.shutdown()}} call {{FileLocalizer.deleteTempFile()}},
but this would cause problems in a multi-threaded environment, since it seems {{ElementDescriptors}}
are pushed onto the {{toDelete}} stack before they're used, not once they're done with. With
this approach, running multiple instances of {{PigServer}} in separate threads could cause
one completed job to clobber the other's still-in-use temp files.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message