hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surenkumar Nihalani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9196) Modify BloomFilter read() and write() to address memory concerns
Date Sun, 20 Jan 2013 06:14:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558169#comment-13558169

Surenkumar Nihalani commented on HADOOP-9196:

James, Turns out the internal variable in {{BitSet}} is private and not protected. So, I can't
have access to it in a child class. 

Do you think it's acceptable to copy paste implementation of {{BitSet}} from JDK into another
class for Hadoop Common?
I know it's violates open/close principle but the nature of your benchmark's statistics suggests
it should be worth it.

What do you think?
> Modify BloomFilter read() and write() to address memory concerns
> ----------------------------------------------------------------
>                 Key: HADOOP-9196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9196
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: James
>            Priority: Minor
> It appears that org.apache.hadoop.util.bloom.BloomFilter's write() method creates a byte
array large enough to fit the entire bit vector into memory during serialization.  This is
unnecessary and may cause out of memory issues if the bit vector is sufficiently large and
memory is tight.   

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message