Mailing-List: contact dev-help@directmemory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@directmemory.apache.org
Received-SPF: pass (athena.apache.org: domain of raffaele.p.guidi@gmail.com
 designates 209.85.215.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <509A208D.3020609@gmail.com>
References: <5099AE92.5060808@gmail.com>
	<CAAnVOG2Gz0T4Gew=CUwZ+0bWd_mF4P50TD-whAjw9mroJHfDhA@mail.gmail.com>
	<509A208D.3020609@gmail.com>
Date: Wed, 7 Nov 2012 17:13:06 +0100
Message-ID: 
 <CAAnVOG3yq1p8DsgMgmTHdf51Cj9inhHZ+wyhy0XC0+zHDLCvRA@mail.gmail.com>
Subject: Re: MapDB
From: "Raffaele P. Guidi" <raffaele.p.guidi@gmail.com>
To: dev@directmemory.apache.org
Content-Type: multipart/alternative; boundary=f46d0407144b6dc0fa04cde9feca

--f46d0407144b6dc0fa04cde9feca
Content-Type: text/plain; charset=ISO-8859-1

>>>> DirectMemory could make good use of mapdb to serialize
least frequently used items to disk
>> The only problem may be that MapDB currently does not support concurrent
transactions (it has only one single global transaction). Not sure if it
could be a problem. However it implements ConcurrentMap, so it is possible
to swap items atomically

We should investigate better on the concurrency theme - thing is that we
have a background thread that does cache eviction and, should we overflow
to disk through jdbm this thread should be the only "writer" - and it would
work perfectly. Of course items would be already serialized (as they are
already off-heap), Is this is allowed by MapDB

>>>>> We could merge our serialization efforts [...]
>> My only condition is that lighting is distributed in separate JAR. I
like minimal dependencies.

Indeed lighting has a separate distribution (and maven artifacts).

>> I would prefer  MapDB to stay on GitHub [...] JDBM3 (previous version)
nearly become ApacheDS subproject, but on last moment I decided otherwise.

We would greatly appreciate your contribution in any case and would be
happy to contribute back as we can :)

Ciao,
     R


On Wed, Nov 7, 2012 at 9:49 AM, Jan Kotek <kjan80@gmail.com> wrote:

> Hi,
>
>      1. DirectMemory could make good use of mapdb to serialize least
>>
>>     frequently used items to disk and free memory
>>     2. DirectMemory could implement a MapDB disk based store in addition
>> to
>>
>>     the bytebuffer and unsafe ones
>>
> The only problem may be that MapDB currently does not support concurrent
> transactions (it has only one single global transaction).
> Not sure if it could be a problem.
>
> However it implements ConcurrentMap, so it is possible to swap items
> atomically
>
>      3. MapDB could take advantage of DM's componentization approach to
>>
>>     support multiple serializers (we believe each one has its advantages
>> in
>>     different scenarios)
>>
> MapDB already supports alternative serializers. User can supply their own
> on Map (similar to table) creation.
> I would love to integrate stuff from lightning serializer.
>
>      4. MapDB could use DM to write items to an off-heap before writing to
>>
>>     disk (asynchronously) to improve speed
>>
> Not sure it would be practical. MapDB already uses memory mapped files so
> effect would be very similar. My tests shows that there is only 50%
> performance difference between inMemory store and onDisk store.
>
> Currently MapDB has only heap based inMemory store. But implementing off
> heap memory store is trivial and I will do it soon.
>
>      5. We could merge our serialization efforts (I believe lightning is
>> very
>>
>>     fast and worth to be considered) and provide an even better solution
>> or two
>>     alternative implementations
>>
>
> 100% agree. I will check lightning sources and see if I could contribute
> my stuff. MapDB serialization is very space-efficiency oriented and it can
> contribute a lot.
>
> My only condition is that lighting is distributed in separate JAR. I like
> minimal dependencies.
>
>
>  In both cases we would be open to contribution in different forms - just
>> contributing patches or with you to join us and the ASF as module or
>> subproject (the latter options have to undergo a formal vote by all
>> project
>> members, of course) as I strongly believe that merging efforts would bring
>> to a better and more complete product.
>>
> I would prefer  MapDB to stay on GitHub.  I find it more comfortable to
> use.
> JDBM3 (previous version) nearly become ApacheDS subproject, but on last
> moment I decided otherwise.
>
>
> Jan
>

--f46d0407144b6dc0fa04cde9feca--