directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raffaele P. Guidi" <raffaele.p.gu...@gmail.com>
Subject Re: Additional Serializer and raw Buffer access
Date Sat, 29 Sep 2012 17:59:41 GMT
Negative part of ASF membership? You get together with a lot of geeky,
talented people with a fixation for software and open source. Oh wait but
this is actually nice! :-D
Il giorno 29/set/2012 19:05, "Olivier Lamy" <olamy@apache.org> ha scritto:

> 2012/9/29 Noctarius <me@noctarius.com>:
> > Thanks Olivier for carify, I'll take a look in it tomorrow but
> > there's just one question left (for now ;)):
> > What is that vote for becoming a committer? What if the vote will
> > be negative?
> The vote is on private list (pmc list for privacy reasons and possible
> negative stuff being on public lists)
> > Until now I just used Apache stuff, was never interested in being
> > part of it so I guess it can be negative for any reason, can't it?
> I don't see why it could be negative but suspens .... :-)
> >
> > Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >> 2012/9/29 Noctarius <me@noctarius.com>:
> >>> Nope my real name is Christoph Engelbert, but Noctarius is
> >>> the all time nick :)
> >>>
> >>> Renaming the package should be no problem, is it
> >>> "org.apache.directmemory.lightning" or what would it be?
> >> fine for me
> >>> Then there needs to be a change in the license header as
> >>> Olivier mentioned, that means just remove the first sentence
> >>> or is there anything more to do (maybe it's easiest thing to
> >>> just copy the header from DM file ;))?
> >> yup use same header as DM
> >>>
> >>> The CLA is just a form to clarify that the source can be
> >>> contributed to the Apache Foundation?
> >> yup correct.
> >>>
> >>> The final step will be attaching the patch in form of a huge
> >>> diff file?
> >> yes
> >>>
> >>> And what is the way to apply for a membership? Never thought
> >>> about how to do that.
> >> Read here http://apache.org/foundation/how-it-works.html and
> >> here http://apache.org/foundation/getinvolved.html . And feel
> >> free to ask any questions :-)
> >>>
> >>> Chris
> >>>
> >>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >>>> OK, deal, at least for me ;-) I propose you rename the
> >>>> packages, produce a patch for this and the new serializer
> >>>> module (should be simple enough starting from an existing
> >>>> one) and, in the meanwhile, apply for ASF membership. Is
> >>>> IP clearance needed? I guess yes. After this we will come
> >>>> up with a formal vote regarding Noctarius (is this your
> >>>> real name?!) allowance in the project team.
> >>>>
> >>>> Good times are gonna come :-)
> >>>>
> >>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
> >>>> <olamy@apache.org> ha scritto:
> >>>>
> >>>>> 2012/9/29 Raffaele P. Guidi
> >>>>> <raffaele.p.guidi@gmail.com>:
> >>>>>> Well we already have a NIO ready interface allowing
> >>>>>> direct access to DMs managed bytebuffers but I think
> >>>>>> this is just half way to what could be achieved
> >>>>>> optimally blending serialization and memory allocation
> >>>>>> together.
> >>>>>>
> >>>>>> Lightning as a module is of course a good idea and it
> >>>>>> could easily evolve as a subproject (for the more
> >>>>>> experienced asf members: is it a feasible way?).
> >>>>> Nothing prevent to have
> >>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>
> >>>>>
> > with a package like: o.a.d.lightning That will be a
> >>>>> subproject.
> >>>>>>
> >>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
> >>>>>> <me@noctarius.com> ha scritto:
> >>>>>>
> >>>>>>> Hey guys,
> >>>>>>>
> >>>>>>> ok there's no lightning binary available since
> >>>>>>> lightning wasn't ready yet for releasing.
> >>>>>>>
> >>>>>>> For being the only developer it would be no problem
> >>>>>>> to contribute the sourcebase for DirectMemory but I'm
> >>>>>>> not sure yet if it wouldn't be better to seperate it
> >>>>>>> to be available without using DirectMemory itself. I
> >>>>>>> started it as a serializer for cluster
> >>>>>>> synchronization, but it would be cool to contribute
> >>>>>>> lightning as a subproject to DirectMemory :-)
> >>>>>>>
> >>>>>>> About the second project I would love to see a
> >>>>>>> public available buffer API directly in DirectMemory
> >>>>>>> so that project would be nearly needless :-) The only
> >>>>>>> difference I think is the allocation strategy my
> >>>>>>> implementation is using against the one DirectMemory
> >>>>>>> has, but I'm pretty sure the allocation is extensible
> >>>>>>> ;-)
> >>>>>>>
> >>>>>>> Like I said, for both projects I'm the only dev so
> >>>>>>> there would be no IP problem. So if it's ok to you to
> >>>>>>> not include lightning directly in DM I would be glad
> >>>>>>> to contribute to the Apache Foundation :-)
> >>>>>>>
> >>>>>>> Cheers Chris
> >>>>>>>
> >>>>>>>
> >>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> >>>>>>>> Ok so it's up to noctarius - your move! ;-)
> >>>>>>>> Regarding the new unsafe storage: it's an opt-in
> >>>>>>>> feature that can be set with the fluent API
> >>>>> (and
> >>>>>>>> soon through the conference file).
> >>>>>>>>
> >>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
> >>>>>>>> Lamy" <olamy@apache.org> ha
> >>>>>>> scritto:
> >>>>>>>>
> >>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>> <raffaele.p.guidi@gmail.com>:
> >>>>>>>>>>>> At least for the moment he can provide
a
> >>>>>>>>>>>> patch to be integrated
> >>>>> in DM
> >>>>>>>>> :-)
> >>>>>>>>>>
> >>>>>>>>>> Sure, but as lightning is not in any public
> >>>>>>>>>> mvn repo should its
> >>>>> code be
> >>>>>>>>>> re-published in our svn? Or what?
> >>>>>>>>> @Apache we don't care about binaries, only
> >>>>>>>>> sources are important ! (a bit theorical for sure
> >>>>>>>>> but that's it :-) ). So if Noctarius was the only
> >>>>>>>>> guy who participate in lightning. He can just
> >>>>>>>>> provide a patch we could integrate as a new dm
> >>>>>>>>> module (note: the patch must not contains any
> >>>>>>>>> more copyright and all sources must have ASF
> >>>>>>>>> licenses).
> >>>>>>>>>
> >>>>>>>>> "Copyright 2012 the original author or authors."
> >>>>>>>>> must be removed. And BTW package must be changed
> >>>>>>>>> :-) (com.github is not acceptable
> >>>>> @asf
> >>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
> >>>>>>>>> )
> >>>>>>>>>
> >>>>>>>>> And having him as a committer will be only a
> >>>>>>>>> matter of voting (we
> >>>>> have
> >>>>>>>>> a great chair who take cares of administrative
> >>>>>>>>> stuff :P )
> >>>>>>>>>
> >>>>>>>>> If some others have participated in the project,
> >>>>>>>>> we must pass tru an ip clearance mechanism
> >>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>
> >>>>>>>>>
> > and all contributors to lightning must provide a cla.
> >>>>>>>>> (It it's the case I can help)
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> perso I'd like we avoid hard dependency
on
> >>>>>>>>>>>> Unsafe as maybe some
> >>>>> use
> >>>>>>>>>> other jdks :-)
> >>>>>>>>>>
> >>>>>>>>>> Well, I believe Unsafe is supported by Sun
> >>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
> >>>>>>>>>> believe that it is more than enough. Also keep
> >>>>>>>>>> in
> >>>>> mind
> >>>>>>>>> that
> >>>>>>>>>> we already have an alternative Unsafe based
> >>>>>>>>>> memory storage - and
> >>>>>>> although
> >>>>>>>>>> it's not thoroughly tested for performance it
> >>>>>>>>>> dramaticaly simplifies
> >>>>>>>>> code,
> >>>>>>>>>> I have great expectations about it.
> >>>>>>>>> Me too :-). Yup definitely more simple and faster
> >>>>>>>>> ! But we must provide a switch off configuration
> >>>>>>>>> mechanism if some
> >>>>> users
> >>>>>>>>> don't want to use that (because Unsafe is just
> >>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
> >>>>>>>>> at your changes with using Unsafe.
> >>>>>>>>>>
> >>>>>>>>>> Cheers, R
> >>>>>>>>>>
> >>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
> >>>>>>>>>> <olamy@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>> <raffaele.p.guidi@gmail.com>:
> >>>>>>>>>>>> What do you think about: 1) implementing
a
> >>>>>>>>>>>> lightning serialization module 2) creating
> >>>>>>>>>>>> a serializer that directly works on
a
> >>>>>>>>>>>> directmemory
> >>>>>>>>> provider
> >>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe
based
> >>>>>>>>>>>> Pointer?
> >>>>>>>>>>>
> >>>>>>>>>>> Sounds good (perso I'd like we avoid hard
> >>>>>>>>>>> dependency on Unsafe as maybe some use other
> >>>>>>>>>>> jdks :-) )
> >>>>>>>>>>>>
> >>>>>>>>>>>> Now I see lightning is apache licensed
and
> >>>>>>>>>>>> this is fine but I
> >>>>> don't
> >>>>>>>>> think
> >>>>>>>>>>>> it is published in any public maven
repo,
> >>>>>>>>>>>> am I right? We could
> >>>>> find a
> >>>>>>>>> way
> >>>>>>>>>>>> to deal with this; options vary from
> >>>>>>>>>>>> publishing lightning to the
> >>>>> free
> >>>>>>>>>>>> sonatype repo,  joining the ASF (which
is
> >>>>>>>>>>>> great anyhow!) and
> >>>>>>>>> republishing
> >>>>>>>>>>>> lightning code in DirectMemory becoming
a
> >>>>>>>>>>>> commiter (which has to
> >>>>>>>>> undergo
> >>>>>>>>>>> a
> >>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>
> >>>>>>>>>>> At least for the moment he can provide a
> >>>>>>>>>>> patch to be integrated in
> >>>>> DM
> >>>>>>>>> :-)
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'd like to hear your and the team feelings
> >>>>>>>>>>>> on this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
> >>>>>>>>>>>> <me@noctarius.com>
> >>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> that's quite similar to what I did
at
> >>>>>>>>>>>>> work. We're developing
> >>>>> Flash
> >>>>>>>>>>>>> online games and using a customized
AMF
> >>>>>>>>>>>>> serialization. To support async
writing
> >>>>>>>>>>>>> of the clients event results I added
> >>>>>>>>>>>>> serialization
> >>>>> of
> >>>>>>>>>>>>> the components / entities to the
players
> >>>>>>>>>>>>> zone calculation and
> >>>>> stored
> >>>>>>>>>>>>> the pre-serialized bytestream directly
to
> >>>>>>>>>>>>> the off-heap (using direct-ring-cache
> >>>>>>>>>>>>> implementation). When the client
> >>>>>>>>>>>>> requests the results (using long-polling)
> >>>>>>>>>>>>> I just write the pre-serialized
> >>>>> data to
> >>>>>>>>>>>>> the right position to deserialize
it by
> >>>>>>>>>>>>> standard ways on Flash
> >>>>> side.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So yeah an seriliaztion to off-heap
> >>>>>>>>>>>>> algorithm would be fine. You
> >>>>> can
> >>>>>>>>>>>>> avoid using byte arrays and minimalize
> >>>>>>>>>>>>> GC.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele
P.
> >>>>>>>>>>>>> Guidi:
> >>>>>>>>>>>>>> Nice to hear back from you!
Yes, I was
> >>>>>>>>>>>>>> thinking about creating a
> >>>>>>>>> new
> >>>>>>>>>>>>> memory
> >>>>>>>>>>>>>> storage implementation using
Unsafe
> >>>>>>>>>>>>>> (and I did it, recently) and
> >>>>>>>>>>> also, as
> >>>>>>>>>>>>>> DirectMemory relies heavily
on
> >>>>>>>>>>>>>> serialization (and supports
many
> >>>>> of
> >>>>>>>>>>> them,
> >>>>>>>>>>>>>> protostuff, protobuf, msgpack
and of
> >>>>>>>>>>>>>> course standard
> >>>>>>>>> serialization),
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>> creating a simple embedded serializer
> >>>>>>>>>>>>>> leveraging the same
> >>>>>>>>> techniques
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> used (Unsafe and bytecode generation).
> >>>>>>>>>>>>>> The idea with embedding is avoiding
to
> >>>>>>>>>>>>>> serialize in a byte array
> >>>>>>>>> and
> >>>>>>>>>>> then
> >>>>>>>>>>>>>> moving the byte array to off-heap
> >>>>>>>>>>>>>> memory (via Unsafe or
> >>>>>>>>> ByteBuffers),
> >>>>>>>>>>> and
> >>>>>>>>>>>>>> serializing directly into a
"managed"
> >>>>>>>>>>>>>> off-heap buffer, thus
> >>>>> further
> >>>>>>>>>>>>>> optimizing heap utilization
(less GC).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think about it?
Does it
> >>>>>>>>>>>>>> make any sense to you?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
PM,
> >>>>>>>>>>>>>> Noctarius <me@noctarius.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Raffaele found out about
a project
> >>>>>>>>>>>>>>> of mine (Lightning) a few
> >>>>> weeks
> >>>>>>>>>>>>>>> ago. Lightning is a heavy
Unsafe and
> >>>>>>>>>>>>>>> Bytecode generation using
Serializer
> >>>>>>>>>>>>>>> implementation. He told
me that he
> >>>>>>>>>>>>>>> was interested in adding
something
> >>>>>>>>>>>>>>> similar to DirectMemory
and I would
> >>>>>>>>>>>>>>> be glad to
> >>>>>>>>> help
> >>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Another project I started
a few days
> >>>>>>>>>>>>>>> ago, since it was needed
> >>>>> for
> >>>>>>>>>>>>>>> work is DirectRingCache.
The name
> >>>>>>>>>>>>>>> not really meets to actual
> >>>>>>>>>>>>>>> implementation since it's
not yet a
> >>>>>>>>>>>>>>> ring buffer using cache.
I
> >>>>>>>>> used
> >>>>>>>>>>>>>>> this for a pre-serialization
simple
> >>>>>>>>>>>>>>> bytestream cache with self-growing
> >>>>>>>>>>>>>>> buffers. It could be nice
to have
> >>>>>>>>>>>>>>> DirectMemory
> >>>>> having
> >>>>>>>>>>>>>>> raw "buffers" to write to
or to read
> >>>>>>>>>>>>>>> from.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Here are the links from
the
> >>>>>>>>>>>>>>> projects:
> >>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> > https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>> It would be nice to help out since I really like the idea of
> >>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>> direct-ring-cache is some
kind of
> >>>>>>>>> reinventing
> >>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>>
> >>>>
> >>
> >>
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message