incubator-directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <>
Subject Re: Involvement as a developer
Date Wed, 19 Oct 2011 06:25:37 GMT
On Wed, Oct 19, 2011 at 11:02 AM, Akash Ashok <> wrote:
> Thanks Guys for responding on a quick reply. HBase is indeed my first love
> :)
> I was really interested in what DirectMemory could offer and am excited to
> see that its pretty awesome. I posed that question because I wasn't really
> sure as to what the rules were on the incubator :)  Great to know I could
> get started right away. Shall start digging thru the code.

All ASF projects work the same way :)

> I went through the road map and have a few questions:
> 1. File Storage -
> This was my main concern. This moves more towards a database approach or to
> be more specific Key Value store approach  Aren't there existing file based
> solutions which can be used instead of developing another one?

Yes there are. We can use any of key-value stores or Object DB's like
BerkleyDB or others.
It would be good to have a pluggable layer, where a persistant
provider can be chosen.
However, its OffHeap aka stored in RAM, so don't feel its a priority,
and persistance would make sense when we evolve into a Cache

> 2. Hazelcast for replication -
> If i get this right the idea to use it more like a replication framework and
> then use that to cache them onto the DirectMemory on the respective systems
> ?

Hazelcast already has released OffHeap storage. Moreover, its a
complete Cache framework with provision of adding persistant stores.
So I am very much against using it. Please note that I use Hazelcast
myself, so not implying that its not a good framework.

Going Distributed is a different story, and if you see the past
conversations, our current focus is on OffHeap store.
To go distributed, we have to move scope from OffHeap store to a Cache
solution. Then comes the challenges Distributed or Replicated.
Replicated is easy, but Distributed world would bring in a lot more
design choice like Server or peer2peer architecture, what concurrency
levels to support etc.

My take
1. Implement OffHeap store
2. Benchmark it, refine it to give latencies that we need in
production (would luv sub-milliseconds or keep then less than 2 ms)

Once we reach here, we can think about going complete caching
solution, followed by going distributed :)

> Lastly "I recently rewrote DM entirely for simplification." +1 on the fact
> of simplification.
> As Einstien once said " Make things as simple as possible, not simpler" :)
> Cheers,
> Akash A
> On Wed, Oct 19, 2011 at 8:33 AM, Raffaele P. Guidi <
>> wrote:
>> Uhm, I'm new in the ASF, but I guess you just need to read some docs (ok,
>> we
>> don't have many, just something in the old wiki @github and an initial
>> roadmap proposal [1]), take a look at the code and the issues and get
>> started :) I see googling around that you have interests in hadoop and
>> hbase
>> as well, so maybe you could start investigating sinergies with those
>> products (just an idea, it is in the roadmap).
>> In any case thanks for your interest and welcome aboard.
>> Ciao,
>>    R
>> [1] Roadmap proposal:
>> On Wed, Oct 19, 2011 at 4:44 AM, Akash Ashok <>
>> wrote:
>> > I know this is still getting incubated. But I would like to get involved
>> in
>> > the development. What are the rules like? Can I get involved ?
>> >
>> > Cheers,
>> > Akash A
>> >


My Photo Galleries:

View raw message