directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noctarius ...@noctarius.com>
Subject Re: Additional Serializer and raw Buffer access
Date Sat, 29 Sep 2012 17:00:30 GMT
Thanks Olivier for carify, I'll take a look in it tomorrow but
there's just one question left (for now ;)):
What is that vote for becoming a committer? What if the vote will
be negative?
Until now I just used Apache stuff, was never interested in being
part of it so I guess it can be negative for any reason, can't it?

Am 29.09.2012 18:56, schrieb Olivier Lamy:
> 2012/9/29 Noctarius <me@noctarius.com>:
>> Nope my real name is Christoph Engelbert, but Noctarius is
>> the all time nick :)
>> 
>> Renaming the package should be no problem, is it 
>> "org.apache.directmemory.lightning" or what would it be?
> fine for me
>> Then there needs to be a change in the license header as
>> Olivier mentioned, that means just remove the first sentence
>> or is there anything more to do (maybe it's easiest thing to
>> just copy the header from DM file ;))?
> yup use same header as DM
>> 
>> The CLA is just a form to clarify that the source can be 
>> contributed to the Apache Foundation?
> yup correct.
>> 
>> The final step will be attaching the patch in form of a huge
>> diff file?
> yes
>> 
>> And what is the way to apply for a membership? Never thought
>> about how to do that.
> Read here http://apache.org/foundation/how-it-works.html and
> here http://apache.org/foundation/getinvolved.html . And feel
> free to ask any questions :-)
>> 
>> Chris
>> 
>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>> OK, deal, at least for me ;-) I propose you rename the 
>>> packages, produce a patch for this and the new serializer 
>>> module (should be simple enough starting from an existing
>>> one) and, in the meanwhile, apply for ASF membership. Is
>>> IP clearance needed? I guess yes. After this we will come
>>> up with a formal vote regarding Noctarius (is this your
>>> real name?!) allowance in the project team.
>>> 
>>> Good times are gonna come :-)
>>> 
>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy" 
>>> <olamy@apache.org> ha scritto:
>>> 
>>>> 2012/9/29 Raffaele P. Guidi
>>>> <raffaele.p.guidi@gmail.com>:
>>>>> Well we already have a NIO ready interface allowing
>>>>> direct access to DMs managed bytebuffers but I think
>>>>> this is just half way to what could be achieved
>>>>> optimally blending serialization and memory allocation
>>>>> together.
>>>>> 
>>>>> Lightning as a module is of course a good idea and it
>>>>> could easily evolve as a subproject (for the more
>>>>> experienced asf members: is it a feasible way?).
>>>> Nothing prevent to have 
>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>
>>>> 
with a package like: o.a.d.lightning That will be a
>>>> subproject.
>>>>> 
>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius" 
>>>>> <me@noctarius.com> ha scritto:
>>>>> 
>>>>>> Hey guys,
>>>>>> 
>>>>>> ok there's no lightning binary available since
>>>>>> lightning wasn't ready yet for releasing.
>>>>>> 
>>>>>> For being the only developer it would be no problem
>>>>>> to contribute the sourcebase for DirectMemory but I'm
>>>>>> not sure yet if it wouldn't be better to seperate it
>>>>>> to be available without using DirectMemory itself. I
>>>>>> started it as a serializer for cluster
>>>>>> synchronization, but it would be cool to contribute
>>>>>> lightning as a subproject to DirectMemory :-)
>>>>>> 
>>>>>> About the second project I would love to see a
>>>>>> public available buffer API directly in DirectMemory
>>>>>> so that project would be nearly needless :-) The only
>>>>>> difference I think is the allocation strategy my
>>>>>> implementation is using against the one DirectMemory
>>>>>> has, but I'm pretty sure the allocation is extensible
>>>>>> ;-)
>>>>>> 
>>>>>> Like I said, for both projects I'm the only dev so
>>>>>> there would be no IP problem. So if it's ok to you to
>>>>>> not include lightning directly in DM I would be glad
>>>>>> to contribute to the Apache Foundation :-)
>>>>>> 
>>>>>> Cheers Chris
>>>>>> 
>>>>>> 
>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>>> Ok so it's up to noctarius - your move! ;-)
>>>>>>> Regarding the new unsafe storage: it's an opt-in
>>>>>>> feature that can be set with the fluent API
>>>> (and
>>>>>>> soon through the conference file).
>>>>>>> 
>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
>>>>>>> Lamy" <olamy@apache.org> ha
>>>>>> scritto:
>>>>>>> 
>>>>>>>> 2012/9/29 Raffaele P. Guidi 
>>>>>>>> <raffaele.p.guidi@gmail.com>:
>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>> patch to be integrated
>>>> in DM
>>>>>>>> :-)
>>>>>>>>> 
>>>>>>>>> Sure, but as lightning is not in any public
>>>>>>>>> mvn repo should its
>>>> code be
>>>>>>>>> re-published in our svn? Or what?
>>>>>>>> @Apache we don't care about binaries, only
>>>>>>>> sources are important ! (a bit theorical for sure
>>>>>>>> but that's it :-) ). So if Noctarius was the only
>>>>>>>> guy who participate in lightning. He can just
>>>>>>>> provide a patch we could integrate as a new dm
>>>>>>>> module (note: the patch must not contains any
>>>>>>>> more copyright and all sources must have ASF
>>>>>>>> licenses).
>>>>>>>> 
>>>>>>>> "Copyright 2012 the original author or authors."
>>>>>>>> must be removed. And BTW package must be changed
>>>>>>>> :-) (com.github is not acceptable
>>>> @asf
>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
>>>>>>>> )
>>>>>>>> 
>>>>>>>> And having him as a committer will be only a
>>>>>>>> matter of voting (we
>>>> have
>>>>>>>> a great chair who take cares of administrative
>>>>>>>> stuff :P )
>>>>>>>> 
>>>>>>>> If some others have participated in the project,
>>>>>>>> we must pass tru an ip clearance mechanism 
>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>
>>>>>>>> 
and all contributors to lightning must provide a cla.
>>>>>>>> (It it's the case I can help)
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> perso I'd like we avoid hard dependency on 
>>>>>>>>>>> Unsafe as maybe some
>>>> use
>>>>>>>>> other jdks :-)
>>>>>>>>> 
>>>>>>>>> Well, I believe Unsafe is supported by Sun
>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
>>>>>>>>> believe that it is more than enough. Also keep
>>>>>>>>> in
>>>> mind
>>>>>>>> that
>>>>>>>>> we already have an alternative Unsafe based
>>>>>>>>> memory storage - and
>>>>>> although
>>>>>>>>> it's not thoroughly tested for performance it 
>>>>>>>>> dramaticaly simplifies
>>>>>>>> code,
>>>>>>>>> I have great expectations about it.
>>>>>>>> Me too :-). Yup definitely more simple and faster
>>>>>>>> ! But we must provide a switch off configuration 
>>>>>>>> mechanism if some
>>>> users
>>>>>>>> don't want to use that (because Unsafe is just 
>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
>>>>>>>> at your changes with using Unsafe.
>>>>>>>>> 
>>>>>>>>> Cheers, R
>>>>>>>>> 
>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy 
>>>>>>>>> <olamy@apache.org>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> 2012/9/29 Raffaele P. Guidi 
>>>>>>>>>> <raffaele.p.guidi@gmail.com>:
>>>>>>>>>>> What do you think about: 1) implementing a 
>>>>>>>>>>> lightning serialization module 2) creating
>>>>>>>>>>> a serializer that directly works on a 
>>>>>>>>>>> directmemory
>>>>>>>> provider
>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based 
>>>>>>>>>>> Pointer?
>>>>>>>>>> 
>>>>>>>>>> Sounds good (perso I'd like we avoid hard 
>>>>>>>>>> dependency on Unsafe as maybe some use other
>>>>>>>>>> jdks :-) )
>>>>>>>>>>> 
>>>>>>>>>>> Now I see lightning is apache licensed and
>>>>>>>>>>> this is fine but I
>>>> don't
>>>>>>>> think
>>>>>>>>>>> it is published in any public maven repo,
>>>>>>>>>>> am I right? We could
>>>> find a
>>>>>>>> way
>>>>>>>>>>> to deal with this; options vary from
>>>>>>>>>>> publishing lightning to the
>>>> free
>>>>>>>>>>> sonatype repo,  joining the ASF (which is
>>>>>>>>>>> great anyhow!) and
>>>>>>>> republishing
>>>>>>>>>>> lightning code in DirectMemory becoming a 
>>>>>>>>>>> commiter (which has to
>>>>>>>> undergo
>>>>>>>>>> a
>>>>>>>>>>> PMC vote).
>>>>>>>>>> 
>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>> patch to be integrated in
>>>> DM
>>>>>>>> :-)
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I'd like to hear your and the team feelings
>>>>>>>>>>> on this.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius 
>>>>>>>>>>> <me@noctarius.com>
>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>> 
>>>>>>>>>>>> that's quite similar to what I did at
>>>>>>>>>>>> work. We're developing
>>>> Flash
>>>>>>>>>>>> online games and using a customized AMF 
>>>>>>>>>>>> serialization. To support async writing
>>>>>>>>>>>> of the clients event results I added 
>>>>>>>>>>>> serialization
>>>> of
>>>>>>>>>>>> the components / entities to the players
>>>>>>>>>>>> zone calculation and
>>>> stored
>>>>>>>>>>>> the pre-serialized bytestream directly to
>>>>>>>>>>>> the off-heap (using direct-ring-cache 
>>>>>>>>>>>> implementation). When the client
>>>>>>>>>>>> requests the results (using long-polling)
>>>>>>>>>>>> I just write the pre-serialized
>>>> data to
>>>>>>>>>>>> the right position to deserialize it by 
>>>>>>>>>>>> standard ways on Flash
>>>> side.
>>>>>>>>>>>> 
>>>>>>>>>>>> So yeah an seriliaztion to off-heap
>>>>>>>>>>>> algorithm would be fine. You
>>>> can
>>>>>>>>>>>> avoid using byte arrays and minimalize
>>>>>>>>>>>> GC.
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.

>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>> Nice to hear back from you! Yes, I was

>>>>>>>>>>>>> thinking about creating a
>>>>>>>> new
>>>>>>>>>>>> memory
>>>>>>>>>>>>> storage implementation using Unsafe
>>>>>>>>>>>>> (and I did it, recently) and
>>>>>>>>>> also, as
>>>>>>>>>>>>> DirectMemory relies heavily on 
>>>>>>>>>>>>> serialization (and supports many
>>>> of
>>>>>>>>>> them,
>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
>>>>>>>>>>>>> course standard
>>>>>>>> serialization),
>>>>>>>>>>>> about
>>>>>>>>>>>>> creating a simple embedded serializer

>>>>>>>>>>>>> leveraging the same
>>>>>>>> techniques
>>>>>>>>>> you
>>>>>>>>>>>>> used (Unsafe and bytecode generation).
>>>>>>>>>>>>> The idea with embedding is avoiding to

>>>>>>>>>>>>> serialize in a byte array
>>>>>>>> and
>>>>>>>>>> then
>>>>>>>>>>>>> moving the byte array to off-heap
>>>>>>>>>>>>> memory (via Unsafe or
>>>>>>>> ByteBuffers),
>>>>>>>>>> and
>>>>>>>>>>>>> serializing directly into a "managed"

>>>>>>>>>>>>> off-heap buffer, thus
>>>> further
>>>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> What do you think about it? Does it
>>>>>>>>>>>>> make any sense to you?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
>>>>>>>>>>>>> Noctarius <me@noctarius.com>
>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Raffaele found out about a project
>>>>>>>>>>>>>> of mine (Lightning) a few
>>>> weeks
>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe
and 
>>>>>>>>>>>>>> Bytecode generation using Serializer

>>>>>>>>>>>>>> implementation. He told me that he
>>>>>>>>>>>>>> was interested in adding something
>>>>>>>>>>>>>> similar to DirectMemory and I would
>>>>>>>>>>>>>> be glad to
>>>>>>>> help
>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Another project I started a few days
>>>>>>>>>>>>>> ago, since it was needed
>>>> for
>>>>>>>>>>>>>> work is DirectRingCache. The name
>>>>>>>>>>>>>> not really meets to actual
>>>>>>>>>>>>>> implementation since it's not yet
a
>>>>>>>>>>>>>> ring buffer using cache. I
>>>>>>>> used
>>>>>>>>>>>>>> this for a pre-serialization simple

>>>>>>>>>>>>>> bytestream cache with self-growing

>>>>>>>>>>>>>> buffers. It could be nice to have

>>>>>>>>>>>>>> DirectMemory
>>>> having
>>>>>>>>>>>>>> raw "buffers" to write to or to read

>>>>>>>>>>>>>> from.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Here are the links from the
>>>>>>>>>>>>>> projects: 
>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>> It would be nice to help out since I really like the idea of
>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>> direct-ring-cache is some kind of
>>>>>>>> reinventing
>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>> http://coders.talend.com 
>>>>>>>>>> http://twitter.com/olamy | 
>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com 
>>>>>>>> http://twitter.com/olamy | 
>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- Olivier Lamy Talend: http://coders.talend.com 
>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>> 
>>> 
> 
> 
> 

Mime
View raw message