directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Lamy <ol...@apache.org>
Subject Re: Additional Serializer and raw Buffer access
Date Sat, 29 Sep 2012 15:58:07 GMT
2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
> Well we already have a NIO ready interface allowing direct access to DMs
> managed bytebuffers but I think this is just half way to what could be
> achieved optimally blending serialization and memory allocation together.
>
> Lightning as a module is of course a good idea and it could easily evolve
> as a subproject (for the more experienced asf members: is it a feasible
> way?).
Nothing prevent to have
http://svn.apache.org/repos/asf/directmemory/lightning/trunk
with a package like: o.a.d.lightning
That will be a subproject.
>
> Ciao,
>     R
> Il giorno 29/set/2012 17:44, "Noctarius" <me@noctarius.com> ha scritto:
>
>> Hey guys,
>>
>> ok there's no lightning binary available since lightning wasn't
>> ready yet for releasing.
>>
>> For being the only developer it would be no problem to contribute
>> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
>> be better to seperate it to be available without using DirectMemory
>> itself. I started it as a serializer for cluster synchronization,
>> but it would be cool to contribute lightning as a subproject to
>> DirectMemory :-)
>>
>> About the second project I would love to see a public available
>> buffer API directly in DirectMemory so that project would be nearly
>> needless :-) The only difference I think is the allocation strategy
>> my implementation is using against the one DirectMemory has, but I'm
>> pretty sure the allocation is extensible ;-)
>>
>> Like I said, for both projects I'm the only dev so there would be no
>> IP problem. So if it's ok to you to not include lightning directly
>> in DM I would be glad to contribute to the Apache Foundation :-)
>>
>> Cheers Chris
>>
>>
>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
>> > storage: it's an opt-in feature that can be set with the fluent API (and
>> > soon through the conference file).
>> >
>> > Ciao,
>> >     R
>> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <olamy@apache.org> ha
>> scritto:
>> >
>> >> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
>> >>>>> At least for the moment he can provide a patch to be integrated
in DM
>> >> :-)
>> >>>
>> >>> Sure, but as lightning is not in any public mvn repo should its code
be
>> >>> re-published in our svn? Or what?
>> >> @Apache we don't care about binaries, only sources are important ! (a
>> >> bit theorical for sure but that's it :-) ).
>> >> So if Noctarius was the only guy who participate in lightning. He can
>> >> just provide a patch we could integrate as a new dm module (note: the
>> >> patch must not contains any more copyright and all sources must have
>> >> ASF licenses).
>> >>
>> >> "Copyright 2012 the original author or authors." must be removed.
>> >> And BTW package must be changed :-) (com.github is not acceptable @asf
>> >> :-) )(@Noctarius are you working for github ? :-) )
>> >>
>> >> And having him as a committer will be only a matter of voting (we have
>> >> a great chair who take cares of administrative stuff :P )
>> >>
>> >> If some others have participated in the project, we must pass tru an
>> >> ip clearance mechanism
>> >> (http://incubator.apache.org/ip-clearance/index.html) and all
>> >> contributors to lightning must provide a cla. (It it's the case I can
>> >> help)
>> >>
>> >>>
>> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some
use
>> >>> other jdks :-)
>> >>>
>> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
>> >>> JRockit - and I believe that it is more than enough. Also keep in mind
>> >> that
>> >>> we already have an alternative Unsafe based memory storage - and
>> although
>> >>> it's not thoroughly tested for performance it dramaticaly simplifies
>> >> code,
>> >>> I have great expectations about it.
>> >> Me too :-). Yup definitely more simple and faster !
>> >> But we must provide a switch off configuration mechanism if some users
>> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
>> >> And sorry I didn't have a look yet at your changes with using Unsafe.
>> >>>
>> >>> Cheers,
>> >>>     R
>> >>>
>> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <olamy@apache.org>
>> wrote:
>> >>>
>> >>>> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
>> >>>>> What do you think about:
>> >>>>> 1) implementing a lightning serialization module
>> >>>>> 2) creating a serializer that directly works on a directmemory
>> >> provider
>> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
>> >>>>
>> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
>> >>>> maybe some use other jdks :-) )
>> >>>>>
>> >>>>> Now I see lightning is apache licensed and this is fine but
I don't
>> >> think
>> >>>>> it is published in any public maven repo, am I right? We could
find a
>> >> way
>> >>>>> to deal with this; options vary from publishing lightning to
the free
>> >>>>> sonatype repo,  joining the ASF (which is great anyhow!) and
>> >> republishing
>> >>>>> lightning code in DirectMemory becoming a commiter (which has
to
>> >> undergo
>> >>>> a
>> >>>>> PMC vote).
>> >>>>
>> >>>> At least for the moment he can provide a patch to be integrated
in DM
>> >> :-)
>> >>>>
>> >>>>>
>> >>>>> I'd like to hear your and the team feelings on this.
>> >>>>>
>> >>>>> Thanks,
>> >>>>>     Raffaele
>> >>>>>
>> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me@noctarius.com>
wrote:
>> >>>>>
>> >>>>>> Hey Raffaele,
>> >>>>>>
>> >>>>>> that's quite similar to what I did at work. We're developing
Flash
>> >>>>>> online games and using a customized AMF serialization. To
support
>> >>>>>> async writing of the clients event results I added serialization
of
>> >>>>>> the components / entities to the players zone calculation
and stored
>> >>>>>> the pre-serialized bytestream directly to the off-heap (using
>> >>>>>> direct-ring-cache implementation). When the client requests
the
>> >>>>>> results (using long-polling) I just write the pre-serialized
data to
>> >>>>>> the right position to deserialize it by standard ways on
Flash side.
>> >>>>>>
>> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine.
You can
>> >>>>>> avoid using byte arrays and minimalize GC.
>> >>>>>>
>> >>>>>> Cheers
>> >>>>>> Chris
>> >>>>>>
>> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>> >>>>>>> Nice to hear back from you! Yes, I was thinking about
creating a
>> >> new
>> >>>>>> memory
>> >>>>>>> storage implementation using Unsafe (and I did it, recently)
and
>> >>>> also, as
>> >>>>>>> DirectMemory relies heavily on serialization (and supports
many of
>> >>>> them,
>> >>>>>>> protostuff, protobuf, msgpack and of course standard
>> >> serialization),
>> >>>>>> about
>> >>>>>>> creating a simple embedded serializer leveraging the
same
>> >> techniques
>> >>>> you
>> >>>>>>> used (Unsafe and bytecode generation).
>> >>>>>>> The idea with embedding is avoiding to serialize in
a byte array
>> >> and
>> >>>> then
>> >>>>>>> moving the byte array to off-heap memory (via Unsafe
or
>> >> ByteBuffers),
>> >>>> and
>> >>>>>>> serializing directly into a "managed" off-heap buffer,
thus further
>> >>>>>>> optimizing heap utilization (less GC).
>> >>>>>>>
>> >>>>>>> What do you think about it? Does it make any sense to
you?
>> >>>>>>>
>> >>>>>>> Ciao,
>> >>>>>>>     R
>> >>>>>>>
>> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me@noctarius.com>
>> >> wrote:
>> >>>>>>>
>> >>>>>>>> Hey guys,
>> >>>>>>>>
>> >>>>>>>> Raffaele found out about a project of mine (Lightning)
a few weeks
>> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation
using
>> >>>>>>>> Serializer implementation. He told me that he was
interested in
>> >>>>>>>> adding something similar to DirectMemory and I would
be glad to
>> >> help
>> >>>>>>>> out in this.
>> >>>>>>>>
>> >>>>>>>> Another project I started a few days ago, since
it was needed for
>> >>>>>>>> work is DirectRingCache. The name not really meets
to actual
>> >>>>>>>> implementation since it's not yet a ring buffer
using cache. I
>> >> used
>> >>>>>>>> this for a pre-serialization simple bytestream cache
with
>> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory
having
>> >>>>>>>> raw "buffers" to write to or to read from.
>> >>>>>>>>
>> >>>>>>>> Here are the links from the projects:
>> >>>>>>>> https://github.com/noctarius/Lightning
>> >>>>>>>> https://github.com/noctarius/direct-ring-cache
>> >>>>>>>>
>> >>>>>>>> It would be nice to help out since I really like
the idea of
>> >>>>>>>> DirectMemory and since direct-ring-cache is some
kind of
>> >> reinventing
>> >>>>>>>> the wheel.
>> >>>>>>>>
>> >>>>>>>> Cheers
>> >>>>>>>> Noctarius (Chris)
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Olivier Lamy
>> >>>> Talend: http://coders.talend.com
>> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >>>>
>> >>
>> >>
>> >>
>> >> --
>> >> Olivier Lamy
>> >> Talend: http://coders.talend.com
>> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >>
>> >
>>



-- 
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy

Mime
View raw message