hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Pullara (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-40) Memory management in BigDataBag is probably wrong
Date Sat, 01 Dec 2007 21:04:43 GMT

    [ https://issues.apache.org/jira/browse/PIG-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547514
] 

Sam Pullara commented on PIG-40:
--------------------------------

I've tested the attached program in a few VMs/OSs.  It appears to do something more like what
you want and it will increase performance as well because memory will only be checked after
automatic garbage collections instead of before every add to the bag.  I haven't yet integrated
it in to PIG as I would like to get any feedback about it. 

> Memory management in BigDataBag is probably wrong
> -------------------------------------------------
>
>                 Key: PIG-40
>                 URL: https://issues.apache.org/jira/browse/PIG-40
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Sam Pullara
>         Attachments: MemoryUsage.java
>
>
> src/org/apache/pig/data/BigDataBag.java
> 1) You should not use finalizers for things other than external resources -- using them
here is very dangerous and could inadvertantly lead to deadlocks and object resurrection and
just decreases performance without any advantage.
> 2) Using .freeMemory() the way it is used in this class is broken.  freeMemory() is going
to return a mostly random number between 0 and the real amount.  Adding gc() in here is a
terrible performance burden.  If you really want to do something like this you should using
softreferences and finalization queues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message