hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Reed <br...@yahoo-inc.com>
Subject Re: [jira] Commented: (PIG-40) Memory management in BigDataBag is probably wrong
Date Thu, 13 Dec 2007 15:44:04 GMT
Yes the spillable objects are data bags. We you say "We don't want bags 
themselves to be GCed" I assume you are referring to the use of weak 
references? I'm using weak references to allow the bags to get GCed when they 
are not used any more. If we implement the MemoryManager without weak 
references you would need an unregister() method or bags would never get 
GCed.

I could add an isMemoryLow() method, but I don't see a need for anyone to use 
it. The MemoryManager will automatically spill the bags when memory gets low 
by calling something equivalent to reclaimMemory().

ben

On Wednesday 12 December 2007 16:22:49 Utkarsh Srivastava wrote:
> The spillable objects are data bags right? We don't want bags
> themselves to be garbage collected. Its the tuples inside them that
> are the problem.
>
> I was thinking more along the lines of
>
> > class MemoryManager {
>
> 	List<Spillable> spillables;
> 	public static MemoryManager getMemoryManager();
>
> >     public void register(Spillable spillable){
>
> 		spillables.add(spillable);
> 	}
> 	public boolean isMemoryLow(){
> 		//Sam's stuff
> 	}
>
> 	private reclaimMemory(){
> 		//traverse the list and somehow ask the biggest one to spill
> 		spillable.spill();
> 		//garbage collect?
> 	}
>
> > }
>
> On Dec 12, 2007, at 3:31 PM, Benjamin Reed (JIRA) wrote:
> >     [ https://issues.apache.org/jira/browse/PIG-40?
> > page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> > tabpanel#action_12551159 ]
> >
> > Benjamin Reed commented on PIG-40:
> > ----------------------------------
> >
> > How about
> >
> > class MemoryManager {
> >     static MemoryManager getMemoryManager();
> >     void add(Spillable spillable);
> > }
> >
> > add() will put Spillable on a WeakList so that the Spillable object
> > will get GCed when strong references go away.
> >
> >> Memory management in BigDataBag is probably wrong
> >> -------------------------------------------------
> >>
> >>                 Key: PIG-40
> >>                 URL: https://issues.apache.org/jira/browse/PIG-40
> >>             Project: Pig
> >>          Issue Type: Bug
> >>          Components: impl
> >>            Reporter: Sam Pullara
> >>            Assignee: Benjamin Reed
> >>         Attachments: BigDataBag.java, MemoryUsage.java
> >>
> >>
> >> src/org/apache/pig/data/BigDataBag.java
> >> 1) You should not use finalizers for things other than external
> >> resources -- using them here is very dangerous and could
> >> inadvertantly lead to deadlocks and object resurrection and just
> >> decreases performance without any advantage.
> >> 2) Using .freeMemory() the way it is used in this class is
> >> broken.  freeMemory() is going to return a mostly random number
> >> between 0 and the real amount.  Adding gc() in here is a terrible
> >> performance burden.  If you really want to do something like this
> >> you should using softreferences and finalization queues.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.



Mime
View raw message