hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich" <ol...@yahoo-inc.com>
Subject RE: [jira] Commented: (PIG-40) Memory management in BigDataBag is probably wrong
Date Thu, 13 Dec 2007 00:37:39 GMT
I like this proposal. I was thinking along the same lines.

Olga 

-----Original Message-----
From: Utkarsh Srivastava [mailto:utkarsh@yahoo-inc.com] 
Sent: Wednesday, December 12, 2007 4:23 PM
To: pig-dev@incubator.apache.org
Subject: Re: [jira] Commented: (PIG-40) Memory management in BigDataBag
is probably wrong

The spillable objects are data bags right? We don't want bags themselves
to be garbage collected. Its the tuples inside them that are the
problem.

I was thinking more along the lines of

> class MemoryManager {
	List<Spillable> spillables;
	public static MemoryManager getMemoryManager();
>     public void register(Spillable spillable){
		spillables.add(spillable);
	}
	public boolean isMemoryLow(){
		//Sam's stuff
	}

	private reclaimMemory(){
		//traverse the list and somehow ask the biggest one to
spill
		spillable.spill();
		//garbage collect?
	}

> }
>


On Dec 12, 2007, at 3:31 PM, Benjamin Reed (JIRA) wrote:

>
>     [ https://issues.apache.org/jira/browse/PIG-40? 
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12551159 ]
>
> Benjamin Reed commented on PIG-40:
> ----------------------------------
>
> How about
>
> class MemoryManager {
>     static MemoryManager getMemoryManager();
>     void add(Spillable spillable);
> }
>
> add() will put Spillable on a WeakList so that the Spillable object 
> will get GCed when strong references go away.
>
>> Memory management in BigDataBag is probably wrong
>> -------------------------------------------------
>>
>>                 Key: PIG-40
>>                 URL: https://issues.apache.org/jira/browse/PIG-40
>>             Project: Pig
>>          Issue Type: Bug
>>          Components: impl
>>            Reporter: Sam Pullara
>>            Assignee: Benjamin Reed
>>         Attachments: BigDataBag.java, MemoryUsage.java
>>
>>
>> src/org/apache/pig/data/BigDataBag.java
>> 1) You should not use finalizers for things other than external 
>> resources -- using them here is very dangerous and could 
>> inadvertantly lead to deadlocks and object resurrection and just 
>> decreases performance without any advantage.
>> 2) Using .freeMemory() the way it is used in this class is broken.  
>> freeMemory() is going to return a mostly random number between 0 and 
>> the real amount.  Adding gc() in here is a terrible performance 
>> burden.  If you really want to do something like this you should 
>> using softreferences and finalization queues.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>


Mime
View raw message