Mailing-List: contact dev-help@directmemory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@directmemory.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAAnVOG2v2wzdEYOunCGsqW_O_4QHQn6THb-46_ZMF=V-X_34rw@mail.gmail.com>
References: <5066EC3F.6040602@noctarius.com>
 <CAAnVOG3HGwrP-mxwtNtitWJTZRHUgFhPE_+XKgBM910-Wjvczg@mail.gmail.com>
 <5066F730.9040905@noctarius.com>
 <CAAnVOG3uyvTm0teK2+CT0yt3eesz4yP-r74bsnWWNsnP_Upa8w@mail.gmail.com>
 <CAPoyBqRyiUOCPqMA1867hCnT=3tWPmy1W0JB_O4-OC4jMe0CJQ@mail.gmail.com>
 <CAAnVOG2v2wzdEYOunCGsqW_O_4QHQn6THb-46_ZMF=V-X_34rw@mail.gmail.com>
From: Olivier Lamy <olamy@apache.org>
Date: Sat, 29 Sep 2012 16:44:48 +0200
Message-ID: 
 <CAPoyBqR8JfdEUjto1fHntYXw9iOyaxy5RenunoT1OTqVcPwpaA@mail.gmail.com>
Subject: Re: Additional Serializer and raw Buffer access
To: dev@directmemory.apache.org
Content-Type: text/plain; charset=ISO-8859-1

2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
>>> At least for the moment he can provide a patch to be integrated in DM :-)
>
> Sure, but as lightning is not in any public mvn repo should its code be
> re-published in our svn? Or what?
@Apache we don't care about binaries, only sources are important ! (a
bit theorical for sure but that's it :-) ).
So if Noctarius was the only guy who participate in lightning. He can
just provide a patch we could integrate as a new dm module (note: the
patch must not contains any more copyright and all sources must have
ASF licenses).

"Copyright 2012 the original author or authors." must be removed.
And BTW package must be changed :-) (com.github is not acceptable @asf
:-) )(@Noctarius are you working for github ? :-) )

And having him as a committer will be only a matter of voting (we have
a great chair who take cares of administrative stuff :P )

If some others have participated in the project, we must pass tru an
ip clearance mechanism
(http://incubator.apache.org/ip-clearance/index.html) and all
contributors to lightning must provide a cla. (It it's the case I can
help)

>
>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
> other jdks :-)
>
> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> JRockit - and I believe that it is more than enough. Also keep in mind that
> we already have an alternative Unsafe based memory storage - and although
> it's not thoroughly tested for performance it dramaticaly simplifies code,
> I have great expectations about it.
Me too :-). Yup definitely more simple and faster !
But we must provide a switch off configuration mechanism if some users
don't want to use that (because Unsafe is just "Unsafe" :-) )
And sorry I didn't have a look yet at your changes with using Unsafe.
>
> Cheers,
>     R
>
> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <olamy@apache.org> wrote:
>
>> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
>> > What do you think about:
>> > 1) implementing a lightning serialization module
>> > 2) creating a serializer that directly works on a directmemory provider
>> > ByteBuffer or (maybe better) Unsafe based Pointer?
>>
>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
>> maybe some use other jdks :-) )
>> >
>> > Now I see lightning is apache licensed and this is fine but I don't think
>> > it is published in any public maven repo, am I right? We could find a way
>> > to deal with this; options vary from publishing lightning to the free
>> > sonatype repo,  joining the ASF (which is great anyhow!) and republishing
>> > lightning code in DirectMemory becoming a commiter (which has to undergo
>> a
>> > PMC vote).
>>
>> At least for the moment he can provide a patch to be integrated in DM :-)
>>
>> >
>> > I'd like to hear your and the team feelings on this.
>> >
>> > Thanks,
>> >     Raffaele
>> >
>> > On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me@noctarius.com> wrote:
>> >
>> >> Hey Raffaele,
>> >>
>> >> that's quite similar to what I did at work. We're developing Flash
>> >> online games and using a customized AMF serialization. To support
>> >> async writing of the clients event results I added serialization of
>> >> the components / entities to the players zone calculation and stored
>> >> the pre-serialized bytestream directly to the off-heap (using
>> >> direct-ring-cache implementation). When the client requests the
>> >> results (using long-polling) I just write the pre-serialized data to
>> >> the right position to deserialize it by standard ways on Flash side.
>> >>
>> >> So yeah an seriliaztion to off-heap algorithm would be fine. You can
>> >> avoid using byte arrays and minimalize GC.
>> >>
>> >> Cheers
>> >> Chris
>> >>
>> >> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>> >> > Nice to hear back from you! Yes, I was thinking about creating a new
>> >> memory
>> >> > storage implementation using Unsafe (and I did it, recently) and
>> also, as
>> >> > DirectMemory relies heavily on serialization (and supports many of
>> them,
>> >> > protostuff, protobuf, msgpack and of course standard serialization),
>> >> about
>> >> > creating a simple embedded serializer leveraging the same techniques
>> you
>> >> > used (Unsafe and bytecode generation).
>> >> > The idea with embedding is avoiding to serialize in a byte array and
>> then
>> >> > moving the byte array to off-heap memory (via Unsafe or ByteBuffers),
>> and
>> >> > serializing directly into a "managed" off-heap buffer, thus further
>> >> > optimizing heap utilization (less GC).
>> >> >
>> >> > What do you think about it? Does it make any sense to you?
>> >> >
>> >> > Ciao,
>> >> >     R
>> >> >
>> >> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me@noctarius.com> wrote:
>> >> >
>> >> >> Hey guys,
>> >> >>
>> >> >> Raffaele found out about a project of mine (Lightning) a few weeks
>> >> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
>> >> >> Serializer implementation. He told me that he was interested in
>> >> >> adding something similar to DirectMemory and I would be glad to help
>> >> >> out in this.
>> >> >>
>> >> >> Another project I started a few days ago, since it was needed for
>> >> >> work is DirectRingCache. The name not really meets to actual
>> >> >> implementation since it's not yet a ring buffer using cache. I used
>> >> >> this for a pre-serialization simple bytestream cache with
>> >> >> self-growing buffers. It could be nice to have DirectMemory having
>> >> >> raw "buffers" to write to or to read from.
>> >> >>
>> >> >> Here are the links from the projects:
>> >> >> https://github.com/noctarius/Lightning
>> >> >> https://github.com/noctarius/direct-ring-cache
>> >> >>
>> >> >> It would be nice to help out since I really like the idea of
>> >> >> DirectMemory and since direct-ring-cache is some kind of reinventing
>> >> >> the wheel.
>> >> >>
>> >> >> Cheers
>> >> >> Noctarius (Chris)
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>


-- 
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy