directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noctarius ...@noctarius.com>
Subject Re: Additional Serializer and raw Buffer access
Date Sat, 29 Sep 2012 16:49:08 GMT
Nope my real name is Christoph Engelbert, but Noctarius is the all
time nick :)

Renaming the package should be no problem, is it
"org.apache.directmemory.lightning" or what would it be?
Then there needs to be a change in the license header as Olivier
mentioned, that means just remove the first sentence or is there
anything more to do (maybe it's easiest thing to just copy the
header from DM file ;))?

The CLA is just a form to clarify that the source can be
contributed to the Apache Foundation?

The final step will be attaching the patch in form of a huge diff
file?

And what is the way to apply for a membership? Never thought about
how to do that.

Chris

Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> OK, deal, at least for me ;-) I propose you rename the
> packages, produce a patch for this and the new serializer
> module (should be simple enough starting from an existing one)
> and, in the meanwhile, apply for ASF membership. Is IP
> clearance needed? I guess yes. After this we will come up with
> a formal vote regarding Noctarius (is this your real name?!)
> allowance in the project team.
> 
> Good times are gonna come :-)
> 
> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
> <olamy@apache.org> ha scritto:
> 
>> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
>>> Well we already have a NIO ready interface allowing direct
>>> access to DMs managed bytebuffers but I think this is just
>>> half way to what could be achieved optimally blending
>>> serialization and memory allocation together.
>>> 
>>> Lightning as a module is of course a good idea and it could
>>> easily evolve as a subproject (for the more experienced asf
>>> members: is it a feasible way?).
>> Nothing prevent to have 
>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk 
>> with a package like: o.a.d.lightning That will be a
>> subproject.
>>> 
>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>> <me@noctarius.com> ha scritto:
>>> 
>>>> Hey guys,
>>>> 
>>>> ok there's no lightning binary available since lightning
>>>> wasn't ready yet for releasing.
>>>> 
>>>> For being the only developer it would be no problem to
>>>> contribute the sourcebase for DirectMemory but I'm not
>>>> sure yet if it wouldn't be better to seperate it to be
>>>> available without using DirectMemory itself. I started it
>>>> as a serializer for cluster synchronization, but it would
>>>> be cool to contribute lightning as a subproject to 
>>>> DirectMemory :-)
>>>> 
>>>> About the second project I would love to see a public
>>>> available buffer API directly in DirectMemory so that
>>>> project would be nearly needless :-) The only difference
>>>> I think is the allocation strategy my implementation is
>>>> using against the one DirectMemory has, but I'm pretty
>>>> sure the allocation is extensible ;-)
>>>> 
>>>> Like I said, for both projects I'm the only dev so there
>>>> would be no IP problem. So if it's ok to you to not
>>>> include lightning directly in DM I would be glad to
>>>> contribute to the Apache Foundation :-)
>>>> 
>>>> Cheers Chris
>>>> 
>>>> 
>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>> Ok so it's up to noctarius - your move! ;-) Regarding
>>>>> the new unsafe storage: it's an opt-in feature that can
>>>>> be set with the fluent API
>> (and
>>>>> soon through the conference file).
>>>>> 
>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier Lamy"
>>>>> <olamy@apache.org> ha
>>>> scritto:
>>>>> 
>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>> <raffaele.p.guidi@gmail.com>:
>>>>>>>>> At least for the moment he can provide a patch
>>>>>>>>> to be integrated
>> in DM
>>>>>> :-)
>>>>>>> 
>>>>>>> Sure, but as lightning is not in any public mvn
>>>>>>> repo should its
>> code be
>>>>>>> re-published in our svn? Or what?
>>>>>> @Apache we don't care about binaries, only sources
>>>>>> are important ! (a bit theorical for sure but that's
>>>>>> it :-) ). So if Noctarius was the only guy who
>>>>>> participate in lightning. He can just provide a patch
>>>>>> we could integrate as a new dm module (note: the 
>>>>>> patch must not contains any more copyright and all
>>>>>> sources must have ASF licenses).
>>>>>> 
>>>>>> "Copyright 2012 the original author or authors." must
>>>>>> be removed. And BTW package must be changed :-)
>>>>>> (com.github is not acceptable
>> @asf
>>>>>> :-) )(@Noctarius are you working for github ? :-) )
>>>>>> 
>>>>>> And having him as a committer will be only a matter
>>>>>> of voting (we
>> have
>>>>>> a great chair who take cares of administrative stuff
>>>>>> :P )
>>>>>> 
>>>>>> If some others have participated in the project, we
>>>>>> must pass tru an ip clearance mechanism 
>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>> and all contributors to lightning must provide a cla.
>>>>>> (It it's the case I can help)
>>>>>> 
>>>>>>> 
>>>>>>>>> perso I'd like we avoid hard dependency on
>>>>>>>>> Unsafe as maybe some
>> use
>>>>>>> other jdks :-)
>>>>>>> 
>>>>>>> Well, I believe Unsafe is supported by Sun JDK,
>>>>>>> OpenJDK, IBM JDK and JRockit - and I believe that
>>>>>>> it is more than enough. Also keep in
>> mind
>>>>>> that
>>>>>>> we already have an alternative Unsafe based memory
>>>>>>> storage - and
>>>> although
>>>>>>> it's not thoroughly tested for performance it
>>>>>>> dramaticaly simplifies
>>>>>> code,
>>>>>>> I have great expectations about it.
>>>>>> Me too :-). Yup definitely more simple and faster ! 
>>>>>> But we must provide a switch off configuration
>>>>>> mechanism if some
>> users
>>>>>> don't want to use that (because Unsafe is just
>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet at
>>>>>> your changes with using Unsafe.
>>>>>>> 
>>>>>>> Cheers, R
>>>>>>> 
>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>> <olamy@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>> <raffaele.p.guidi@gmail.com>:
>>>>>>>>> What do you think about: 1) implementing a
>>>>>>>>> lightning serialization module 2) creating a
>>>>>>>>> serializer that directly works on a
>>>>>>>>> directmemory
>>>>>> provider
>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>> Pointer?
>>>>>>>> 
>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>> dependency on Unsafe as maybe some use other jdks
>>>>>>>> :-) )
>>>>>>>>> 
>>>>>>>>> Now I see lightning is apache licensed and this
>>>>>>>>> is fine but I
>> don't
>>>>>> think
>>>>>>>>> it is published in any public maven repo, am I
>>>>>>>>> right? We could
>> find a
>>>>>> way
>>>>>>>>> to deal with this; options vary from publishing
>>>>>>>>> lightning to the
>> free
>>>>>>>>> sonatype repo,  joining the ASF (which is great
>>>>>>>>> anyhow!) and
>>>>>> republishing
>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>> commiter (which has to
>>>>>> undergo
>>>>>>>> a
>>>>>>>>> PMC vote).
>>>>>>>> 
>>>>>>>> At least for the moment he can provide a patch to
>>>>>>>> be integrated in
>> DM
>>>>>> :-)
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I'd like to hear your and the team feelings on
>>>>>>>>> this.
>>>>>>>>> 
>>>>>>>>> Thanks, Raffaele
>>>>>>>>> 
>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>>> <me@noctarius.com>
>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Raffaele,
>>>>>>>>>> 
>>>>>>>>>> that's quite similar to what I did at work.
>>>>>>>>>> We're developing
>> Flash
>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>> serialization. To support async writing of
>>>>>>>>>> the clients event results I added
>>>>>>>>>> serialization
>> of
>>>>>>>>>> the components / entities to the players zone
>>>>>>>>>> calculation and
>> stored
>>>>>>>>>> the pre-serialized bytestream directly to the
>>>>>>>>>> off-heap (using direct-ring-cache
>>>>>>>>>> implementation). When the client requests
>>>>>>>>>> the results (using long-polling) I just write
>>>>>>>>>> the pre-serialized
>> data to
>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>> standard ways on Flash
>> side.
>>>>>>>>>> 
>>>>>>>>>> So yeah an seriliaztion to off-heap algorithm
>>>>>>>>>> would be fine. You
>> can
>>>>>>>>>> avoid using byte arrays and minimalize GC.
>>>>>>>>>> 
>>>>>>>>>> Cheers Chris
>>>>>>>>>> 
>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>>> Guidi:
>>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>>> thinking about creating a
>>>>>> new
>>>>>>>>>> memory
>>>>>>>>>>> storage implementation using Unsafe (and I
>>>>>>>>>>> did it, recently) and
>>>>>>>> also, as
>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>> serialization (and supports many
>> of
>>>>>>>> them,
>>>>>>>>>>> protostuff, protobuf, msgpack and of course
>>>>>>>>>>> standard
>>>>>> serialization),
>>>>>>>>>> about
>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>> leveraging the same
>>>>>> techniques
>>>>>>>> you
>>>>>>>>>>> used (Unsafe and bytecode generation). The
>>>>>>>>>>> idea with embedding is avoiding to
>>>>>>>>>>> serialize in a byte array
>>>>>> and
>>>>>>>> then
>>>>>>>>>>> moving the byte array to off-heap memory
>>>>>>>>>>> (via Unsafe or
>>>>>> ByteBuffers),
>>>>>>>> and
>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>> off-heap buffer, thus
>> further
>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>> 
>>>>>>>>>>> What do you think about it? Does it make
>>>>>>>>>>> any sense to you?
>>>>>>>>>>> 
>>>>>>>>>>> Ciao, R
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius
>>>>>>>>>>> <me@noctarius.com>
>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>> 
>>>>>>>>>>>> Raffaele found out about a project of
>>>>>>>>>>>> mine (Lightning) a few
>> weeks
>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
>>>>>>>>>>>> Bytecode generation using Serializer
>>>>>>>>>>>> implementation. He told me that he was
>>>>>>>>>>>> interested in adding something similar to
>>>>>>>>>>>> DirectMemory and I would be glad to
>>>>>> help
>>>>>>>>>>>> out in this.
>>>>>>>>>>>> 
>>>>>>>>>>>> Another project I started a few days ago,
>>>>>>>>>>>> since it was needed
>> for
>>>>>>>>>>>> work is DirectRingCache. The name not
>>>>>>>>>>>> really meets to actual implementation
>>>>>>>>>>>> since it's not yet a ring buffer using
>>>>>>>>>>>> cache. I
>>>>>> used
>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>> DirectMemory
>> having
>>>>>>>>>>>> raw "buffers" to write to or to read
>>>>>>>>>>>> from.
>>>>>>>>>>>> 
>>>>>>>>>>>> Here are the links from the projects: 
>>>>>>>>>>>> https://github.com/noctarius/Lightning 
>>>>>>>>>>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 
It would be nice to help out since I really like the idea of
>>>>>>>>>>>> DirectMemory and since direct-ring-cache
>>>>>>>>>>>> is some kind of
>>>>>> reinventing
>>>>>>>>>>>> the wheel.
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com 
>>>>>>>> http://twitter.com/olamy |
>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- Olivier Lamy Talend: http://coders.talend.com 
>>>>>> http://twitter.com/olamy |
>>>>>> http://linkedin.com/in/olamy
>>>>>> 
>>>>> 
>>>> 
>> 
>> 
>> 
>> -- Olivier Lamy Talend: http://coders.talend.com 
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> 
> 

Mime
View raw message