hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Graham (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1313) PigServer leaks memory over time
Date Mon, 29 Mar 2010 20:40:27 GMT

     [ https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Bill Graham updated PIG-1313:
-----------------------------

    Attachment: PIG-1313-0.4.0-1.patch
                PIG-1313-1.patch

Attaching two patches, one built from the trunk and one built from the 0.4.0 branch (since
0.4.0 is what we're using). Not sure if patches to older branches are still being applied
or not though. Let me know if I should provide other patches for other branches.

The patch includes the following changes to FileLocalizer:

- toDelete, deleteOnFail and relativeRoot are ThreadLocals.
- The initialized attribute and the init method has been removed, since they're no longer
needed.

In PigServer I've done the following:

- FileLocalizer.deleteTempFiles() is called from the shutdown method.

I've run my server with this patch applied for a few days now and the memory consumption has
stabilized. I also no longer see exceptions in the logs from when a PigServers temp files
would go missing.

> PigServer leaks memory over time
> --------------------------------
>
>                 Key: PIG-1313
>                 URL: https://issues.apache.org/jira/browse/PIG-1313
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Bill Graham
>            Assignee: Bill Graham
>         Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch, Pig1313Reproducer.java
>
>
> When {{PigServer}} runs it creates temporary files using the {{FileLocalizer.getTemporaryPath(..)}}.
This static method creates and returns a handle to a temporary file (as an instance of {{ElementDescriptor}}).
The {{ElementDescriptors}} returned by this method are kept on a static {{Stack}} named {{toDelete}}.
The items on {{toDelete}} get removed by the {{FileLocalizer.deleteTempFile()}} method.
> The only place in the code where I see {{FileLocalizer.deleteTempFile()}} called is in
the Main class. {{PigServer}} does not call that method though, so a long-running VM that
repeatedly uses instances of {{PigServer}} to run jobs will leak memory via {{toDelete}}.
> One suggested fix is to have {{PigServer.shutdown()}} call {{FileLocalizer.deleteTempFile()}},
but this would cause problems in a multi-threaded environment, since it seems {{ElementDescriptors}}
are pushed onto the {{toDelete}} stack before they're used, not once they're done with. With
this approach, running multiple instances of {{PigServer}} in separate threads could cause
one completed job to clobber the other's still-in-use temp files.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message