directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Kotek <kja...@gmail.com>
Subject Re: MapDB
Date Thu, 08 Nov 2012 08:21:37 GMT
Hi,


 > 30 buffers.
JVM kept crashing if I increased memory too much. This was only solution 
that gave me lot of memory and did not crash JVM. Not sure why

 > 0.2 snapshot
You have outdated documentation then. It shows 0.1 (and fragment names 
are wrong BTW).

http://directmemory.apache.org/simple-usage.html

 > Serialization
How can I use better serialization?

 > JVM settings
I left all at default, except increasing max heap (-Xmx4G for DM, 
-Xmx13G for TreeMap).
Are there better settings I should use?

Jan


On 08/11/12 06:44, Raffaele P. Guidi wrote:
> Looking at the code I see Jan implemented against v0.1 which is a lot
> outdated. I would strongly suggest Jan to use 0.2-SNAPSHOT (yeah, I know
> snapshots suck...) and us to work on a new release (it's about time)
> focusing a bit more on performance instead of features. What do you guys
> think?
>
> Ciao,
>      R
>
>
> On Thu, Nov 8, 2012 at 6:16 AM, Raffaele P. Guidi <
> raffaele.p.guidi@gmail.com> wrote:
>
>> Oh, sorry I didn't notice the github link (it's 5 'o clock in the morning
>> here :-D ). Also why thirty 1mb buffers?
>> Il giorno 08/nov/2012 06:11, "Christoph Engelbert" <noctarius@apache.org>
>> ha scritto:
>>
>> Unsafe allocator alone won't speed it up that much because I guess he
>>> relied on standard Serialization for his test.
>>>
>>>
>>>
>>> "Raffaele P. Guidi" <raffaele.p.guidi@gmail.com> schrieb:
>>>
>>>> I don't believe integers or even 1kb buffers are the right case for
>>>> directmemory. Small keys, large payloads (+4kb) are. In any case: how
>>>> were
>>>> the heap and xxmaxdirectmemory settings? And did you try the Unsafe
>>>> storage
>>>> as well? It would be helpful if you could share the benchmark code.
>>>>
>>>> Thanks,
>>>>     R
>>>> Il giorno 08/nov/2012 02:49, "Jan Kotek" <kjan80@gmail.com> ha scritto:
>>>>
>>>>> Hi,
>>>>>
>>>>> MapDB now has DirectMemory storage.
>>>>> I made little test to compare our two libraries + java.util.TreeMap
>>>>>
>>>>> All tests are on 64bit Linux, 16GB RAM, JDK6. Default JVM settings.
>>>>> Source code is here: https://github.com/jankotek/**
>>>>>
>>>> mapdb-junk/tree/master/src/**main/java/**directMemoryBenchmark<
>>> https://github.com/jankotek/mapdb-junk/tree/master/src/main/java/directMemoryBenchmark
>>>>>
>>>>> First is performance test. How long does it take to insert and fetch
>>>> 1
>>>>> million records, each 1KB large.
>>>>>   - DM - 6901 ms
>>>>>   - MapDB - 2045 ms
>>>>>   - java.util.TreeMap - 940 ms
>>>>>
>>>>> Second is overhead test. It inserts pair of integers in cycle, until
>>>> JVM
>>>>> runs out of memory.
>>>>> It should measure how much memory library each consumes for its own
>>>>> structures.
>>>>>   - DM - inserts 9 000 000 records and then it becomes very slow.
>>>> Profiler
>>>>> shows high GC activity.
>>>>>   - MapDB - works about 10 minutes and than it runs out of memory with
>>>> 217
>>>>> 000 000 records.
>>>>>   - java.util.TreeMap - crashes after 99 000 000 records , it consumed
>>>>> about 12GB heap.
>>>>>
>>>>>
>>>>> I am not sure I configured DM correctly, JVM kept crashing when I
>>>>> increased size too much.
>>>>> I ended with this setting:
>>>>>
>>>>>         CacheService<Object, Object> m = new DirectMemory<Object,
>>>> Object>()
>>>>>                  .setNumberOfBuffers( 30 )
>>>>>                  .setSize((int) 1e8)
>>>>>                  .setInitialCapacity( 10000 )
>>>>>                  .setConcurrencyLevel( 4 )
>>>>>                  .setDisposalTime(1000000)
>>>>>                  .newCacheService();
>>>>>
>>>>>
>>>>> Hope this helps.
>>>>>
>>>>> Jan
>>>>>
>>>>>
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>> gesendet.
>>


Mime
View raw message