directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raffaele P. Guidi" <raffaele.p.gu...@gmail.com>
Subject Re: Additional Serializer and raw Buffer access
Date Sat, 29 Sep 2012 16:23:30 GMT
OK, deal, at least for me ;-) I propose you rename the packages, produce a
patch for this and the new serializer module (should be simple enough
starting from an existing one) and, in the meanwhile, apply for ASF
membership. Is IP clearance needed? I guess yes.
After this we will come up with a formal vote regarding Noctarius (is this
your real name?!) allowance in the project team.

Good times are gonna come :-)

Thanks,
    R
Il giorno 29/set/2012 17:58, "Olivier Lamy" <olamy@apache.org> ha scritto:

> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
> > Well we already have a NIO ready interface allowing direct access to DMs
> > managed bytebuffers but I think this is just half way to what could be
> > achieved optimally blending serialization and memory allocation together.
> >
> > Lightning as a module is of course a good idea and it could easily evolve
> > as a subproject (for the more experienced asf members: is it a feasible
> > way?).
> Nothing prevent to have
> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> with a package like: o.a.d.lightning
> That will be a subproject.
> >
> > Ciao,
> >     R
> > Il giorno 29/set/2012 17:44, "Noctarius" <me@noctarius.com> ha scritto:
> >
> >> Hey guys,
> >>
> >> ok there's no lightning binary available since lightning wasn't
> >> ready yet for releasing.
> >>
> >> For being the only developer it would be no problem to contribute
> >> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
> >> be better to seperate it to be available without using DirectMemory
> >> itself. I started it as a serializer for cluster synchronization,
> >> but it would be cool to contribute lightning as a subproject to
> >> DirectMemory :-)
> >>
> >> About the second project I would love to see a public available
> >> buffer API directly in DirectMemory so that project would be nearly
> >> needless :-) The only difference I think is the allocation strategy
> >> my implementation is using against the one DirectMemory has, but I'm
> >> pretty sure the allocation is extensible ;-)
> >>
> >> Like I said, for both projects I'm the only dev so there would be no
> >> IP problem. So if it's ok to you to not include lightning directly
> >> in DM I would be glad to contribute to the Apache Foundation :-)
> >>
> >> Cheers Chris
> >>
> >>
> >> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> >> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
> >> > storage: it's an opt-in feature that can be set with the fluent API
> (and
> >> > soon through the conference file).
> >> >
> >> > Ciao,
> >> >     R
> >> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <olamy@apache.org> ha
> >> scritto:
> >> >
> >> >> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
> >> >>>>> At least for the moment he can provide a patch to be integrated
> in DM
> >> >> :-)
> >> >>>
> >> >>> Sure, but as lightning is not in any public mvn repo should its
> code be
> >> >>> re-published in our svn? Or what?
> >> >> @Apache we don't care about binaries, only sources are important !
(a
> >> >> bit theorical for sure but that's it :-) ).
> >> >> So if Noctarius was the only guy who participate in lightning. He can
> >> >> just provide a patch we could integrate as a new dm module (note: the
> >> >> patch must not contains any more copyright and all sources must have
> >> >> ASF licenses).
> >> >>
> >> >> "Copyright 2012 the original author or authors." must be removed.
> >> >> And BTW package must be changed :-) (com.github is not acceptable
> @asf
> >> >> :-) )(@Noctarius are you working for github ? :-) )
> >> >>
> >> >> And having him as a committer will be only a matter of voting (we
> have
> >> >> a great chair who take cares of administrative stuff :P )
> >> >>
> >> >> If some others have participated in the project, we must pass tru an
> >> >> ip clearance mechanism
> >> >> (http://incubator.apache.org/ip-clearance/index.html) and all
> >> >> contributors to lightning must provide a cla. (It it's the case I can
> >> >> help)
> >> >>
> >> >>>
> >> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe
some
> use
> >> >>> other jdks :-)
> >> >>>
> >> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK
and
> >> >>> JRockit - and I believe that it is more than enough. Also keep
in
> mind
> >> >> that
> >> >>> we already have an alternative Unsafe based memory storage - and
> >> although
> >> >>> it's not thoroughly tested for performance it dramaticaly simplifies
> >> >> code,
> >> >>> I have great expectations about it.
> >> >> Me too :-). Yup definitely more simple and faster !
> >> >> But we must provide a switch off configuration mechanism if some
> users
> >> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
> >> >> And sorry I didn't have a look yet at your changes with using Unsafe.
> >> >>>
> >> >>> Cheers,
> >> >>>     R
> >> >>>
> >> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <olamy@apache.org>
> >> wrote:
> >> >>>
> >> >>>> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
> >> >>>>> What do you think about:
> >> >>>>> 1) implementing a lightning serialization module
> >> >>>>> 2) creating a serializer that directly works on a directmemory
> >> >> provider
> >> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
> >> >>>>
> >> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe
as
> >> >>>> maybe some use other jdks :-) )
> >> >>>>>
> >> >>>>> Now I see lightning is apache licensed and this is fine
but I
> don't
> >> >> think
> >> >>>>> it is published in any public maven repo, am I right? We
could
> find a
> >> >> way
> >> >>>>> to deal with this; options vary from publishing lightning
to the
> free
> >> >>>>> sonatype repo,  joining the ASF (which is great anyhow!)
and
> >> >> republishing
> >> >>>>> lightning code in DirectMemory becoming a commiter (which
has to
> >> >> undergo
> >> >>>> a
> >> >>>>> PMC vote).
> >> >>>>
> >> >>>> At least for the moment he can provide a patch to be integrated
in
> DM
> >> >> :-)
> >> >>>>
> >> >>>>>
> >> >>>>> I'd like to hear your and the team feelings on this.
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>>     Raffaele
> >> >>>>>
> >> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me@noctarius.com>
> wrote:
> >> >>>>>
> >> >>>>>> Hey Raffaele,
> >> >>>>>>
> >> >>>>>> that's quite similar to what I did at work. We're developing
> Flash
> >> >>>>>> online games and using a customized AMF serialization.
To support
> >> >>>>>> async writing of the clients event results I added
serialization
> of
> >> >>>>>> the components / entities to the players zone calculation
and
> stored
> >> >>>>>> the pre-serialized bytestream directly to the off-heap
(using
> >> >>>>>> direct-ring-cache implementation). When the client
requests the
> >> >>>>>> results (using long-polling) I just write the pre-serialized
> data to
> >> >>>>>> the right position to deserialize it by standard ways
on Flash
> side.
> >> >>>>>>
> >> >>>>>> So yeah an seriliaztion to off-heap algorithm would
be fine. You
> can
> >> >>>>>> avoid using byte arrays and minimalize GC.
> >> >>>>>>
> >> >>>>>> Cheers
> >> >>>>>> Chris
> >> >>>>>>
> >> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >> >>>>>>> Nice to hear back from you! Yes, I was thinking
about creating a
> >> >> new
> >> >>>>>> memory
> >> >>>>>>> storage implementation using Unsafe (and I did
it, recently) and
> >> >>>> also, as
> >> >>>>>>> DirectMemory relies heavily on serialization (and
supports many
> of
> >> >>>> them,
> >> >>>>>>> protostuff, protobuf, msgpack and of course standard
> >> >> serialization),
> >> >>>>>> about
> >> >>>>>>> creating a simple embedded serializer leveraging
the same
> >> >> techniques
> >> >>>> you
> >> >>>>>>> used (Unsafe and bytecode generation).
> >> >>>>>>> The idea with embedding is avoiding to serialize
in a byte array
> >> >> and
> >> >>>> then
> >> >>>>>>> moving the byte array to off-heap memory (via Unsafe
or
> >> >> ByteBuffers),
> >> >>>> and
> >> >>>>>>> serializing directly into a "managed" off-heap
buffer, thus
> further
> >> >>>>>>> optimizing heap utilization (less GC).
> >> >>>>>>>
> >> >>>>>>> What do you think about it? Does it make any sense
to you?
> >> >>>>>>>
> >> >>>>>>> Ciao,
> >> >>>>>>>     R
> >> >>>>>>>
> >> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me@noctarius.com>
> >> >> wrote:
> >> >>>>>>>
> >> >>>>>>>> Hey guys,
> >> >>>>>>>>
> >> >>>>>>>> Raffaele found out about a project of mine
(Lightning) a few
> weeks
> >> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode
generation using
> >> >>>>>>>> Serializer implementation. He told me that
he was interested in
> >> >>>>>>>> adding something similar to DirectMemory and
I would be glad to
> >> >> help
> >> >>>>>>>> out in this.
> >> >>>>>>>>
> >> >>>>>>>> Another project I started a few days ago, since
it was needed
> for
> >> >>>>>>>> work is DirectRingCache. The name not really
meets to actual
> >> >>>>>>>> implementation since it's not yet a ring buffer
using cache. I
> >> >> used
> >> >>>>>>>> this for a pre-serialization simple bytestream
cache with
> >> >>>>>>>> self-growing buffers. It could be nice to have
DirectMemory
> having
> >> >>>>>>>> raw "buffers" to write to or to read from.
> >> >>>>>>>>
> >> >>>>>>>> Here are the links from the projects:
> >> >>>>>>>> https://github.com/noctarius/Lightning
> >> >>>>>>>> https://github.com/noctarius/direct-ring-cache
> >> >>>>>>>>
> >> >>>>>>>> It would be nice to help out since I really
like the idea of
> >> >>>>>>>> DirectMemory and since direct-ring-cache is
some kind of
> >> >> reinventing
> >> >>>>>>>> the wheel.
> >> >>>>>>>>
> >> >>>>>>>> Cheers
> >> >>>>>>>> Noctarius (Chris)
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Olivier Lamy
> >> >>>> Talend: http://coders.talend.com
> >> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >> >>>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Olivier Lamy
> >> >> Talend: http://coders.talend.com
> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >> >>
> >> >
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message