directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tatu Saloranta <tsalora...@gmail.com>
Subject Re: KryoSerializer
Date Thu, 01 Nov 2012 17:35:44 GMT
On Thu, Nov 1, 2012 at 10:06 AM, Christoph Engelbert
<noctarius@apache.org> wrote:
> Am 01.11.2012 17:48, schrieb Roman Levenstein:
>> Hi,
>>
>> On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <tsaloranta@gmail.com> wrote:
>>> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
>>> <noctarius@apache.org> wrote:
>>>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>>>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>>>>> <noctarius@apache.org> wrote:
>>>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>>>>> You may want to see how serializer/deserializer in
>>>>>>>
>>>>>>> https://github.com/eishay/jvm-serializers
>>>>>>>
>>>>>>> is done -- Martin wrote it (or at least modified & has maintained
it),
>>>>>>> so it should be along best practices.
>>>>>>>
>>>>>>> -+ Tatu +-
>>>>>> This is only best practice for non concurrent access. The unittests
>>>>>> are executed one by one so the implementation using one byte array
>>>>>> is not a problem but this cannot be threadsafe.
>>>>>>
>>>>>> This applies to this one as well. Never use this implementation in
>>>>>> concurrent environments :-)
>>>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>>>>> Correct. What I meant was it was right way to do it for the use case
>>>>> in question. Which granted requires one to understand the use case...
>>>>> I forgot that this particular test does not do multi-threaded
>>>>> processing like others do.
>>>>>
>>>>> It actually is one of the things for jvm-serializers to consider;
>>>>> efficient reuse techniques vary a lot between codecs.
>>>>>
>>>>> -+ Tatu +-
>>>> I guess there's still the problem about the kryo instance itself. On
>>>> googlecode it is marked that the kryo instance itself is not
>>>> threadsafe. Anyone has some more informations about that? Tatu? :-)
>>> No, I have only tried to use it without success once or twice, outside
>>> of jvm-serializers.
>>> My weapon of choice if Smile+Jakcson. :-D
>>>
>>> But Kryo author (Nate something?) should know -- I can try to dig
>>> contact info, point him to this mailing list.
>>>
>>> -+ Tatu +-
>> I'm one of Kryo's contributors.
>> Kryo instances do keep some context information during serialization
>> and deserialization. Therefore, the same instance cannot be safely
>> used at the same time with multiple input or output streams for
>> performing simultaneous (de) serializations. And using the same
>> instance with multiple threads can result in even more problems.
>>
>> Therefore, when I need to use Kryo in multi-threaded environment, I
>> usually make sure that each thread gets its own instance of Kryo at
>> least for the duration of the operation. The reduce the cost of Kryo
>> instances creation I usually use a pool of such instances. Every time
>> a thread needs to perform a (de) serialization operation, it would get
>> an instance from the pool, perform the operation and return the
>> instance back to the pool. Alternatively, one could use ThreadLocal
>> Kryo instances if applicable.
>>
>> -Roman
>
> Ok that is a nice information. So I would suggest to use
> concurrencylevel to instantiate a bunch of kryo instances and queue
> them. If non of the preinstantiated kryo instances is available than
> the (de-)serialization must wait until one's free again (but that
> would mean the concurrencylevel is choosen to low ;-)).

Instead of queue which needs synchronization, it may be better to use
ThreadLocal with soft reference. This requires no synchronization for
access, and also cleans up instances if needed for GC. This can make a
big difference for multi-threaded access due to reduced lock
contestion.
And the details can be hidden behind a shared factory if that simplifies things.

-+ Tatu +-

Mime
View raw message