directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raffaele P. Guidi" <raffaele.p.gu...@gmail.com>
Subject Re: MapDB
Date Wed, 07 Nov 2012 16:13:06 GMT
>>>> DirectMemory could make good use of mapdb to serialize
least frequently used items to disk
>> The only problem may be that MapDB currently does not support concurrent
transactions (it has only one single global transaction). Not sure if it
could be a problem. However it implements ConcurrentMap, so it is possible
to swap items atomically

We should investigate better on the concurrency theme - thing is that we
have a background thread that does cache eviction and, should we overflow
to disk through jdbm this thread should be the only "writer" - and it would
work perfectly. Of course items would be already serialized (as they are
already off-heap), Is this is allowed by MapDB

>>>>> We could merge our serialization efforts [...]
>> My only condition is that lighting is distributed in separate JAR. I
like minimal dependencies.

Indeed lighting has a separate distribution (and maven artifacts).

>> I would prefer  MapDB to stay on GitHub [...] JDBM3 (previous version)
nearly become ApacheDS subproject, but on last moment I decided otherwise.

We would greatly appreciate your contribution in any case and would be
happy to contribute back as we can :)

Ciao,
     R


On Wed, Nov 7, 2012 at 9:49 AM, Jan Kotek <kjan80@gmail.com> wrote:

> Hi,
>
>      1. DirectMemory could make good use of mapdb to serialize least
>>
>>     frequently used items to disk and free memory
>>     2. DirectMemory could implement a MapDB disk based store in addition
>> to
>>
>>     the bytebuffer and unsafe ones
>>
> The only problem may be that MapDB currently does not support concurrent
> transactions (it has only one single global transaction).
> Not sure if it could be a problem.
>
> However it implements ConcurrentMap, so it is possible to swap items
> atomically
>
>      3. MapDB could take advantage of DM's componentization approach to
>>
>>     support multiple serializers (we believe each one has its advantages
>> in
>>     different scenarios)
>>
> MapDB already supports alternative serializers. User can supply their own
> on Map (similar to table) creation.
> I would love to integrate stuff from lightning serializer.
>
>      4. MapDB could use DM to write items to an off-heap before writing to
>>
>>     disk (asynchronously) to improve speed
>>
> Not sure it would be practical. MapDB already uses memory mapped files so
> effect would be very similar. My tests shows that there is only 50%
> performance difference between inMemory store and onDisk store.
>
> Currently MapDB has only heap based inMemory store. But implementing off
> heap memory store is trivial and I will do it soon.
>
>      5. We could merge our serialization efforts (I believe lightning is
>> very
>>
>>     fast and worth to be considered) and provide an even better solution
>> or two
>>     alternative implementations
>>
>
> 100% agree. I will check lightning sources and see if I could contribute
> my stuff. MapDB serialization is very space-efficiency oriented and it can
> contribute a lot.
>
> My only condition is that lighting is distributed in separate JAR. I like
> minimal dependencies.
>
>
>  In both cases we would be open to contribution in different forms - just
>> contributing patches or with you to join us and the ASF as module or
>> subproject (the latter options have to undergo a formal vote by all
>> project
>> members, of course) as I strongly believe that merging efforts would bring
>> to a better and more complete product.
>>
> I would prefer  MapDB to stay on GitHub.  I find it more comfortable to
> use.
> JDBM3 (previous version) nearly become ApacheDS subproject, but on last
> moment I decided otherwise.
>
>
> Jan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message