incubator-directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raffaele P. Guidi" <>
Subject Re: Memory Deallocation Issue
Date Thu, 20 Oct 2011 14:46:34 GMT
Guys, you are both right - the current allocation strategy has far superior
performance than per-item allocation - but not being able to de-allocate
memory is of course bad in real life (and in a always-on/hot deployment
environment such as OSGi is even worse). This change (great job finding it)
will allow us to add a dispose() or finalize() method that will give back
memory to the O/S between deploys making it more stable and safe to use in
enterprise setups :)


On Thursday, October 20, 2011, Ashish <> wrote:
> On Thu, Oct 20, 2011 at 2:32 PM, Akash Ashok <>
>> On Thu, Oct 20, 2011 at 2:13 PM, Ashish <> wrote:
>>> On Thu, Oct 20, 2011 at 1:57 PM, Raffaele P. Guidi
>>> <> wrote:
>>> > Gooood news! As with every not-so-well documented piece of software I
>>> should
>>> > have read the code before taking wrong assumptions (or at least take a
>>> look
>>> > at stackoverflow ;) ). I think we should ask our mentors to assign
>>> developer
>>> > rights. Or is it to be filed to INFRA? Sorry, I'm still an ASF rookie
>>> >
>>> > Thanks,
>>> >    Raffaele
>>> I again have to disagree on this feature. Why would you need to have
>>> to deallocate memory? You should know how much you need.
>>> Its always better to have a contiguous memory allocated. It works
>>> well. Dynamically resizing will pose challenges and add to performance
>>> issues.
>>> From cache perspective, we are anyways clearing the element, making
>>> way for new element.
>>> IMHO, I see offheap as a big chunk of memory which is pre-allocated
>>> and the MemoryManager we have written should deal with all the object
>>> management. We can manage memory as chunks or realize maps in native
>>> memory, is all upto the design we choose.
>>> This is very much an importand feature I believe. Assume in production
>> made the
>> wrong decision of how much memory was pre-allocated, then you shouldn't
>> charged
>> with the penalty of being unable to use that memory right ? Even though
>> is very expensive
>> the feature should be available but documented well enough, warning
>> its use.
> Well as far as I handle Ops, this is not how things work.
> Ops goes through a detailed capacity planning. Even before Ops, things
> are tested in staging environment.
> So when I take caches to production, I always calculate
> 1. How many elements do I need to store
> 2. What's the average size of each element and extrapolate to how much
> memory is needed
> 3. What level to set the eviction and how much to evict
> 4. Do I need expiry
> 5. For read-only caches, don't want eviction to happen, so tune them
> Need less to mention the GC tuning part.
> Again this is not hard written rule. We all have our preferences &
> experiences.
> From an end user perspective, I want to use DirectMemory so
> a. need a stable release
> b. need some benchmark numbers
> And I am not against this feature, so please go ahead and implement it :)
> I feel we can take an approach of benchmarking these and write
> recommendations to the wiki. wdyt?
>> If you are concerned about memory fragmentation, It wouldn't lead to a
>> of fragmentaion
>> if we deallocate and re allocate contiguous blocks right. I am under the
>>  assumption that
>>  allocateDirect allocates contiguous blocks of memory.
> It tried to allocate. Say if I ask for 64G of direct memory, it shall
> try to allocate contiguous memory.
> Now if we allocate and deallocate, it may not be contiguous. As OS
> can't predict when next allotment request shall come in.
>> @ashok - You don't need dev access to submit patches. Create a JIRA
>>> account and submit patches. New committers @ASF are voted by PMC based
>>> on their contribution :)
>>> I never asked to add me as a committer :) I asked for contributor access
>> where in
>> I have the permission to assign a ticket to myself and SubmitPatch option
>> JIRA.
>> On HBase only after I was added as a contributor I got these options.
>> was the
>> rationale behind my request.
> Ahh.. my mistake.. Sorry ..
> cheers
> ashish

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message