directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noctarius ...@noctarius.com>
Subject Re: Additional Serializer and raw Buffer access
Date Sat, 29 Sep 2012 16:02:47 GMT
If that is possible there's no problem from my side. It would be
an honor to give the Apache Foundation something instead of only
taking code :-)

PS: I'm pretty sure there's a way to make lightning again a little
bit faster by removing boxing on primitives by using
invokedynamic. I started it but left the idea behind to finish the
public api first.

Am 29.09.2012 17:58, schrieb Olivier Lamy:
> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
>> Well we already have a NIO ready interface allowing direct
>> access to DMs managed bytebuffers but I think this is just
>> half way to what could be achieved optimally blending
>> serialization and memory allocation together.
>> 
>> Lightning as a module is of course a good idea and it could
>> easily evolve as a subproject (for the more experienced asf
>> members: is it a feasible way?).
> Nothing prevent to have 
> http://svn.apache.org/repos/asf/directmemory/lightning/trunk 
> with a package like: o.a.d.lightning That will be a
> subproject.
>> 
>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>> <me@noctarius.com> ha scritto:
>> 
>>> Hey guys,
>>> 
>>> ok there's no lightning binary available since lightning
>>> wasn't ready yet for releasing.
>>> 
>>> For being the only developer it would be no problem to
>>> contribute the sourcebase for DirectMemory but I'm not sure
>>> yet if it wouldn't be better to seperate it to be available
>>> without using DirectMemory itself. I started it as a
>>> serializer for cluster synchronization, but it would be
>>> cool to contribute lightning as a subproject to 
>>> DirectMemory :-)
>>> 
>>> About the second project I would love to see a public
>>> available buffer API directly in DirectMemory so that
>>> project would be nearly needless :-) The only difference I
>>> think is the allocation strategy my implementation is using
>>> against the one DirectMemory has, but I'm pretty sure the
>>> allocation is extensible ;-)
>>> 
>>> Like I said, for both projects I'm the only dev so there
>>> would be no IP problem. So if it's ok to you to not include
>>> lightning directly in DM I would be glad to contribute to
>>> the Apache Foundation :-)
>>> 
>>> Cheers Chris
>>> 
>>> 
>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>> Ok so it's up to noctarius - your move! ;-) Regarding the
>>>> new unsafe storage: it's an opt-in feature that can be
>>>> set with the fluent API (and soon through the conference
>>>> file).
>>>> 
>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier Lamy"
>>>> <olamy@apache.org> ha
>>> scritto:
>>>> 
>>>>> 2012/9/29 Raffaele P. Guidi
>>>>> <raffaele.p.guidi@gmail.com>:
>>>>>>>> At least for the moment he can provide a patch to
>>>>>>>> be integrated in DM
>>>>> :-)
>>>>>> 
>>>>>> Sure, but as lightning is not in any public mvn repo
>>>>>> should its code be re-published in our svn? Or what?
>>>>> @Apache we don't care about binaries, only sources are
>>>>> important ! (a bit theorical for sure but that's it :-)
>>>>> ). So if Noctarius was the only guy who participate in
>>>>> lightning. He can just provide a patch we could
>>>>> integrate as a new dm module (note: the patch must not
>>>>> contains any more copyright and all sources must have 
>>>>> ASF licenses).
>>>>> 
>>>>> "Copyright 2012 the original author or authors." must
>>>>> be removed. And BTW package must be changed :-)
>>>>> (com.github is not acceptable @asf :-) )(@Noctarius are
>>>>> you working for github ? :-) )
>>>>> 
>>>>> And having him as a committer will be only a matter of
>>>>> voting (we have a great chair who take cares of
>>>>> administrative stuff :P )
>>>>> 
>>>>> If some others have participated in the project, we
>>>>> must pass tru an ip clearance mechanism 
>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>> and all contributors to lightning must provide a cla.
>>>>> (It it's the case I can help)
>>>>> 
>>>>>> 
>>>>>>>> perso I'd like we avoid hard dependency on Unsafe
>>>>>>>> as maybe some use
>>>>>> other jdks :-)
>>>>>> 
>>>>>> Well, I believe Unsafe is supported by Sun JDK,
>>>>>> OpenJDK, IBM JDK and JRockit - and I believe that it
>>>>>> is more than enough. Also keep in mind
>>>>> that
>>>>>> we already have an alternative Unsafe based memory
>>>>>> storage - and
>>> although
>>>>>> it's not thoroughly tested for performance it
>>>>>> dramaticaly simplifies
>>>>> code,
>>>>>> I have great expectations about it.
>>>>> Me too :-). Yup definitely more simple and faster ! But
>>>>> we must provide a switch off configuration mechanism if
>>>>> some users don't want to use that (because Unsafe is
>>>>> just "Unsafe" :-) ) And sorry I didn't have a look yet
>>>>> at your changes with using Unsafe.
>>>>>> 
>>>>>> Cheers, R
>>>>>> 
>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>> <olamy@apache.org>
>>> wrote:
>>>>>> 
>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>> <raffaele.p.guidi@gmail.com>:
>>>>>>>> What do you think about: 1) implementing a
>>>>>>>> lightning serialization module 2) creating a
>>>>>>>> serializer that directly works on a directmemory
>>>>> provider
>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>> Pointer?
>>>>>>> 
>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>> dependency on Unsafe as maybe some use other jdks
>>>>>>> :-) )
>>>>>>>> 
>>>>>>>> Now I see lightning is apache licensed and this
>>>>>>>> is fine but I don't
>>>>> think
>>>>>>>> it is published in any public maven repo, am I
>>>>>>>> right? We could find a
>>>>> way
>>>>>>>> to deal with this; options vary from publishing
>>>>>>>> lightning to the free sonatype repo,  joining the
>>>>>>>> ASF (which is great anyhow!) and
>>>>> republishing
>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>> commiter (which has to
>>>>> undergo
>>>>>>> a
>>>>>>>> PMC vote).
>>>>>>> 
>>>>>>> At least for the moment he can provide a patch to
>>>>>>> be integrated in DM
>>>>> :-)
>>>>>>> 
>>>>>>>> 
>>>>>>>> I'd like to hear your and the team feelings on
>>>>>>>> this.
>>>>>>>> 
>>>>>>>> Thanks, Raffaele
>>>>>>>> 
>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>> <me@noctarius.com> wrote:
>>>>>>>> 
>>>>>>>>> Hey Raffaele,
>>>>>>>>> 
>>>>>>>>> that's quite similar to what I did at work.
>>>>>>>>> We're developing Flash online games and using a
>>>>>>>>> customized AMF serialization. To support async
>>>>>>>>> writing of the clients event results I added
>>>>>>>>> serialization of the components / entities to
>>>>>>>>> the players zone calculation and stored the
>>>>>>>>> pre-serialized bytestream directly to the
>>>>>>>>> off-heap (using direct-ring-cache
>>>>>>>>> implementation). When the client requests the 
>>>>>>>>> results (using long-polling) I just write the
>>>>>>>>> pre-serialized data to the right position to
>>>>>>>>> deserialize it by standard ways on Flash side.
>>>>>>>>> 
>>>>>>>>> So yeah an seriliaztion to off-heap algorithm
>>>>>>>>> would be fine. You can avoid using byte arrays
>>>>>>>>> and minimalize GC.
>>>>>>>>> 
>>>>>>>>> Cheers Chris
>>>>>>>>> 
>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>> Guidi:
>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>> thinking about creating a
>>>>> new
>>>>>>>>> memory
>>>>>>>>>> storage implementation using Unsafe (and I
>>>>>>>>>> did it, recently) and
>>>>>>> also, as
>>>>>>>>>> DirectMemory relies heavily on serialization
>>>>>>>>>> (and supports many of
>>>>>>> them,
>>>>>>>>>> protostuff, protobuf, msgpack and of course
>>>>>>>>>> standard
>>>>> serialization),
>>>>>>>>> about
>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>> leveraging the same
>>>>> techniques
>>>>>>> you
>>>>>>>>>> used (Unsafe and bytecode generation). The
>>>>>>>>>> idea with embedding is avoiding to serialize
>>>>>>>>>> in a byte array
>>>>> and
>>>>>>> then
>>>>>>>>>> moving the byte array to off-heap memory (via
>>>>>>>>>> Unsafe or
>>>>> ByteBuffers),
>>>>>>> and
>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>> off-heap buffer, thus further optimizing heap
>>>>>>>>>> utilization (less GC).
>>>>>>>>>> 
>>>>>>>>>> What do you think about it? Does it make any
>>>>>>>>>> sense to you?
>>>>>>>>>> 
>>>>>>>>>> Ciao, R
>>>>>>>>>> 
>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius
>>>>>>>>>> <me@noctarius.com>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey guys,
>>>>>>>>>>> 
>>>>>>>>>>> Raffaele found out about a project of mine
>>>>>>>>>>> (Lightning) a few weeks ago. Lightning is a
>>>>>>>>>>> heavy Unsafe and Bytecode generation using 
>>>>>>>>>>> Serializer implementation. He told me that
>>>>>>>>>>> he was interested in adding something
>>>>>>>>>>> similar to DirectMemory and I would be glad
>>>>>>>>>>> to
>>>>> help
>>>>>>>>>>> out in this.
>>>>>>>>>>> 
>>>>>>>>>>> Another project I started a few days ago,
>>>>>>>>>>> since it was needed for work is
>>>>>>>>>>> DirectRingCache. The name not really meets
>>>>>>>>>>> to actual implementation since it's not yet
>>>>>>>>>>> a ring buffer using cache. I
>>>>> used
>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>> bytestream cache with self-growing buffers.
>>>>>>>>>>> It could be nice to have DirectMemory
>>>>>>>>>>> having raw "buffers" to write to or to read
>>>>>>>>>>> from.
>>>>>>>>>>> 
>>>>>>>>>>> Here are the links from the projects: 
>>>>>>>>>>> https://github.com/noctarius/Lightning 
>>>>>>>>>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 
It would be nice to help out since I really like the idea of
>>>>>>>>>>> DirectMemory and since direct-ring-cache is
>>>>>>>>>>> some kind of
>>>>> reinventing
>>>>>>>>>>> the wheel.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com 
>>>>>>> http://twitter.com/olamy |
>>>>>>> http://linkedin.com/in/olamy
>>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- Olivier Lamy Talend: http://coders.talend.com 
>>>>> http://twitter.com/olamy |
>>>>> http://linkedin.com/in/olamy
>>>>> 
>>>> 
>>> 
> 
> 
> 

Mime
View raw message