incubator-directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Memory Deallocation Issue
Date Thu, 20 Oct 2011 09:43:58 GMT
On Thu, Oct 20, 2011 at 2:32 PM, Akash Ashok <thehellmaker@gmail.com> wrote:
> On Thu, Oct 20, 2011 at 2:13 PM, Ashish <paliwalashish@gmail.com> wrote:
>
>> On Thu, Oct 20, 2011 at 1:57 PM, Raffaele P. Guidi
>> <raffaele.p.guidi@gmail.com> wrote:
>> > Gooood news! As with every not-so-well documented piece of software I
>> should
>> > have read the code before taking wrong assumptions (or at least take a
>> look
>> > at stackoverflow ;) ). I think we should ask our mentors to assign
>> developer
>> > rights. Or is it to be filed to INFRA? Sorry, I'm still an ASF rookie ;)
>> >
>> > Thanks,
>> >    Raffaele
>>
>> I again have to disagree on this feature. Why would you need to have
>> to deallocate memory? You should know how much you need.
>> Its always better to have a contiguous memory allocated. It works
>> well. Dynamically resizing will pose challenges and add to performance
>> issues.
>>
>> From cache perspective, we are anyways clearing the element, making
>> way for new element.
>>
>> IMHO, I see offheap as a big chunk of memory which is pre-allocated
>> and the MemoryManager we have written should deal with all the object
>> management. We can manage memory as chunks or realize maps in native
>> memory, is all upto the design we choose.
>>
>> This is very much an importand feature I believe. Assume in production you
> made the
> wrong decision of how much memory was pre-allocated, then you shouldn't be
> charged
> with the penalty of being unable to use that memory right ? Even though this
> is very expensive
> the feature should be available but documented well enough, warning against
> its use.

Well as far as I handle Ops, this is not how things work.
Ops goes through a detailed capacity planning. Even before Ops, things
are tested in staging environment.

So when I take caches to production, I always calculate
1. How many elements do I need to store
2. What's the average size of each element and extrapolate to how much
memory is needed
3. What level to set the eviction and how much to evict
4. Do I need expiry
5. For read-only caches, don't want eviction to happen, so tune them accordingly

Need less to mention the GC tuning part.

Again this is not hard written rule. We all have our preferences &
experiences.

>From an end user perspective, I want to use DirectMemory so
a. need a stable release
b. need some benchmark numbers

And I am not against this feature, so please go ahead and implement it :)
I feel we can take an approach of benchmarking these and write
recommendations to the wiki. wdyt?

>
> If you are concerned about memory fragmentation, It wouldn't lead to a lot
> of fragmentaion
> if we deallocate and re allocate contiguous blocks right. I am under the
>  assumption that
>  allocateDirect allocates contiguous blocks of memory.

It tried to allocate. Say if I ask for 64G of direct memory, it shall
try to allocate contiguous memory.
Now if we allocate and deallocate, it may not be contiguous. As OS
can't predict when next allotment request shall come in.

>
>
> @ashok - You don't need dev access to submit patches. Create a JIRA
>> account and submit patches. New committers @ASF are voted by PMC based
>> on their contribution :)
>>
>> I never asked to add me as a committer :) I asked for contributor access
> where in
> I have the permission to assign a ticket to myself and SubmitPatch option on
> JIRA.
> On HBase only after I was added as a contributor I got these options. This
> was the
> rationale behind my request.
>

Ahh.. my mistake.. Sorry ..

cheers
ashish

Mime
View raw message