Mailing-List: contact dev-help@directmemory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@directmemory.apache.org
Received-SPF: pass (athena.apache.org: domain of raffaele.p.guidi@gmail.com
 designates 74.125.82.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPoyBqR8JfdEUjto1fHntYXw9iOyaxy5RenunoT1OTqVcPwpaA@mail.gmail.com>
References: <5066EC3F.6040602@noctarius.com>
	<CAAnVOG3HGwrP-mxwtNtitWJTZRHUgFhPE_+XKgBM910-Wjvczg@mail.gmail.com>
	<5066F730.9040905@noctarius.com>
	<CAAnVOG3uyvTm0teK2+CT0yt3eesz4yP-r74bsnWWNsnP_Upa8w@mail.gmail.com>
	<CAPoyBqRyiUOCPqMA1867hCnT=3tWPmy1W0JB_O4-OC4jMe0CJQ@mail.gmail.com>
	<CAAnVOG2v2wzdEYOunCGsqW_O_4QHQn6THb-46_ZMF=V-X_34rw@mail.gmail.com>
	<CAPoyBqR8JfdEUjto1fHntYXw9iOyaxy5RenunoT1OTqVcPwpaA@mail.gmail.com>
Date: Sat, 29 Sep 2012 16:53:11 +0200
Message-ID: 
 <CAAnVOG1iFR54qNU1kdVZuF_RFt72vf=24n6i9bziQbnOTag82w@mail.gmail.com>
Subject: Re: Additional Serializer and raw Buffer access
From: "Raffaele P. Guidi" <raffaele.p.guidi@gmail.com>
To: dev@directmemory.apache.org
Content-Type: multipart/alternative; boundary=f46d04428c26cffad204cad85460

--f46d04428c26cffad204cad85460
Content-Type: text/plain; charset=ISO-8859-1

Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
storage: it's an opt-in feature that can be set with the fluent API (and
soon through the conference file).

Ciao,
    R
Il giorno 29/set/2012 16:45, "Olivier Lamy" <olamy@apache.org> ha scritto:

> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
> >>> At least for the moment he can provide a patch to be integrated in DM
> :-)
> >
> > Sure, but as lightning is not in any public mvn repo should its code be
> > re-published in our svn? Or what?
> @Apache we don't care about binaries, only sources are important ! (a
> bit theorical for sure but that's it :-) ).
> So if Noctarius was the only guy who participate in lightning. He can
> just provide a patch we could integrate as a new dm module (note: the
> patch must not contains any more copyright and all sources must have
> ASF licenses).
>
> "Copyright 2012 the original author or authors." must be removed.
> And BTW package must be changed :-) (com.github is not acceptable @asf
> :-) )(@Noctarius are you working for github ? :-) )
>
> And having him as a committer will be only a matter of voting (we have
> a great chair who take cares of administrative stuff :P )
>
> If some others have participated in the project, we must pass tru an
> ip clearance mechanism
> (http://incubator.apache.org/ip-clearance/index.html) and all
> contributors to lightning must provide a cla. (It it's the case I can
> help)
>
> >
> >>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
> > other jdks :-)
> >
> > Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> > JRockit - and I believe that it is more than enough. Also keep in mind
> that
> > we already have an alternative Unsafe based memory storage - and although
> > it's not thoroughly tested for performance it dramaticaly simplifies
> code,
> > I have great expectations about it.
> Me too :-). Yup definitely more simple and faster !
> But we must provide a switch off configuration mechanism if some users
> don't want to use that (because Unsafe is just "Unsafe" :-) )
> And sorry I didn't have a look yet at your changes with using Unsafe.
> >
> > Cheers,
> >     R
> >
> > On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <olamy@apache.org> wrote:
> >
> >> 2012/9/29 Raffaele P. Guidi <raffaele.p.guidi@gmail.com>:
> >> > What do you think about:
> >> > 1) implementing a lightning serialization module
> >> > 2) creating a serializer that directly works on a directmemory
> provider
> >> > ByteBuffer or (maybe better) Unsafe based Pointer?
> >>
> >> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
> >> maybe some use other jdks :-) )
> >> >
> >> > Now I see lightning is apache licensed and this is fine but I don't
> think
> >> > it is published in any public maven repo, am I right? We could find a
> way
> >> > to deal with this; options vary from publishing lightning to the free
> >> > sonatype repo,  joining the ASF (which is great anyhow!) and
> republishing
> >> > lightning code in DirectMemory becoming a commiter (which has to
> undergo
> >> a
> >> > PMC vote).
> >>
> >> At least for the moment he can provide a patch to be integrated in DM
> :-)
> >>
> >> >
> >> > I'd like to hear your and the team feelings on this.
> >> >
> >> > Thanks,
> >> >     Raffaele
> >> >
> >> > On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me@noctarius.com> wrote:
> >> >
> >> >> Hey Raffaele,
> >> >>
> >> >> that's quite similar to what I did at work. We're developing Flash
> >> >> online games and using a customized AMF serialization. To support
> >> >> async writing of the clients event results I added serialization of
> >> >> the components / entities to the players zone calculation and stored
> >> >> the pre-serialized bytestream directly to the off-heap (using
> >> >> direct-ring-cache implementation). When the client requests the
> >> >> results (using long-polling) I just write the pre-serialized data to
> >> >> the right position to deserialize it by standard ways on Flash side.
> >> >>
> >> >> So yeah an seriliaztion to off-heap algorithm would be fine. You can
> >> >> avoid using byte arrays and minimalize GC.
> >> >>
> >> >> Cheers
> >> >> Chris
> >> >>
> >> >> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >> >> > Nice to hear back from you! Yes, I was thinking about creating a
> new
> >> >> memory
> >> >> > storage implementation using Unsafe (and I did it, recently) and
> >> also, as
> >> >> > DirectMemory relies heavily on serialization (and supports many of
> >> them,
> >> >> > protostuff, protobuf, msgpack and of course standard
> serialization),
> >> >> about
> >> >> > creating a simple embedded serializer leveraging the same
> techniques
> >> you
> >> >> > used (Unsafe and bytecode generation).
> >> >> > The idea with embedding is avoiding to serialize in a byte array
> and
> >> then
> >> >> > moving the byte array to off-heap memory (via Unsafe or
> ByteBuffers),
> >> and
> >> >> > serializing directly into a "managed" off-heap buffer, thus further
> >> >> > optimizing heap utilization (less GC).
> >> >> >
> >> >> > What do you think about it? Does it make any sense to you?
> >> >> >
> >> >> > Ciao,
> >> >> >     R
> >> >> >
> >> >> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me@noctarius.com>
> wrote:
> >> >> >
> >> >> >> Hey guys,
> >> >> >>
> >> >> >> Raffaele found out about a project of mine (Lightning) a few weeks
> >> >> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >> >> >> Serializer implementation. He told me that he was interested in
> >> >> >> adding something similar to DirectMemory and I would be glad to
> help
> >> >> >> out in this.
> >> >> >>
> >> >> >> Another project I started a few days ago, since it was needed for
> >> >> >> work is DirectRingCache. The name not really meets to actual
> >> >> >> implementation since it's not yet a ring buffer using cache. I
> used
> >> >> >> this for a pre-serialization simple bytestream cache with
> >> >> >> self-growing buffers. It could be nice to have DirectMemory having
> >> >> >> raw "buffers" to write to or to read from.
> >> >> >>
> >> >> >> Here are the links from the projects:
> >> >> >> https://github.com/noctarius/Lightning
> >> >> >> https://github.com/noctarius/direct-ring-cache
> >> >> >>
> >> >> >> It would be nice to help out since I really like the idea of
> >> >> >> DirectMemory and since direct-ring-cache is some kind of
> reinventing
> >> >> >> the wheel.
> >> >> >>
> >> >> >> Cheers
> >> >> >> Noctarius (Chris)
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >> --
> >> Olivier Lamy
> >> Talend: http://coders.talend.com
> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>

--f46d04428c26cffad204cad85460--