Return-Path: X-Original-To: apmail-directmemory-dev-archive@www.apache.org Delivered-To: apmail-directmemory-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EB2F2D7AD for ; Sat, 29 Sep 2012 15:58:28 +0000 (UTC) Received: (qmail 94768 invoked by uid 500); 29 Sep 2012 15:58:28 -0000 Delivered-To: apmail-directmemory-dev-archive@directmemory.apache.org Received: (qmail 94729 invoked by uid 500); 29 Sep 2012 15:58:28 -0000 Mailing-List: contact dev-help@directmemory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@directmemory.apache.org Delivered-To: mailing list dev@directmemory.apache.org Received: (qmail 94721 invoked by uid 99); 29 Sep 2012 15:58:28 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Sep 2012 15:58:28 +0000 Received: from localhost (HELO mail-ie0-f178.google.com) (127.0.0.1) (smtp-auth username olamy, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Sep 2012 15:58:28 +0000 Received: by ieje11 with SMTP id e11so11121928iej.37 for ; Sat, 29 Sep 2012 08:58:27 -0700 (PDT) Received: by 10.50.188.225 with SMTP id gd1mr1646881igc.15.1348934307944; Sat, 29 Sep 2012 08:58:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.115.215 with HTTP; Sat, 29 Sep 2012 08:58:07 -0700 (PDT) In-Reply-To: References: <5066EC3F.6040602@noctarius.com> <5066F730.9040905@noctarius.com> <506716DD.3080700@noctarius.com> From: Olivier Lamy Date: Sat, 29 Sep 2012 17:58:07 +0200 Message-ID: Subject: Re: Additional Serializer and raw Buffer access To: dev@directmemory.apache.org Content-Type: text/plain; charset=ISO-8859-1 2012/9/29 Raffaele P. Guidi : > Well we already have a NIO ready interface allowing direct access to DMs > managed bytebuffers but I think this is just half way to what could be > achieved optimally blending serialization and memory allocation together. > > Lightning as a module is of course a good idea and it could easily evolve > as a subproject (for the more experienced asf members: is it a feasible > way?). Nothing prevent to have http://svn.apache.org/repos/asf/directmemory/lightning/trunk with a package like: o.a.d.lightning That will be a subproject. > > Ciao, > R > Il giorno 29/set/2012 17:44, "Noctarius" ha scritto: > >> Hey guys, >> >> ok there's no lightning binary available since lightning wasn't >> ready yet for releasing. >> >> For being the only developer it would be no problem to contribute >> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't >> be better to seperate it to be available without using DirectMemory >> itself. I started it as a serializer for cluster synchronization, >> but it would be cool to contribute lightning as a subproject to >> DirectMemory :-) >> >> About the second project I would love to see a public available >> buffer API directly in DirectMemory so that project would be nearly >> needless :-) The only difference I think is the allocation strategy >> my implementation is using against the one DirectMemory has, but I'm >> pretty sure the allocation is extensible ;-) >> >> Like I said, for both projects I'm the only dev so there would be no >> IP problem. So if it's ok to you to not include lightning directly >> in DM I would be glad to contribute to the Apache Foundation :-) >> >> Cheers Chris >> >> >> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi: >> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe >> > storage: it's an opt-in feature that can be set with the fluent API (and >> > soon through the conference file). >> > >> > Ciao, >> > R >> > Il giorno 29/set/2012 16:45, "Olivier Lamy" ha >> scritto: >> > >> >> 2012/9/29 Raffaele P. Guidi : >> >>>>> At least for the moment he can provide a patch to be integrated in DM >> >> :-) >> >>> >> >>> Sure, but as lightning is not in any public mvn repo should its code be >> >>> re-published in our svn? Or what? >> >> @Apache we don't care about binaries, only sources are important ! (a >> >> bit theorical for sure but that's it :-) ). >> >> So if Noctarius was the only guy who participate in lightning. He can >> >> just provide a patch we could integrate as a new dm module (note: the >> >> patch must not contains any more copyright and all sources must have >> >> ASF licenses). >> >> >> >> "Copyright 2012 the original author or authors." must be removed. >> >> And BTW package must be changed :-) (com.github is not acceptable @asf >> >> :-) )(@Noctarius are you working for github ? :-) ) >> >> >> >> And having him as a committer will be only a matter of voting (we have >> >> a great chair who take cares of administrative stuff :P ) >> >> >> >> If some others have participated in the project, we must pass tru an >> >> ip clearance mechanism >> >> (http://incubator.apache.org/ip-clearance/index.html) and all >> >> contributors to lightning must provide a cla. (It it's the case I can >> >> help) >> >> >> >>> >> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use >> >>> other jdks :-) >> >>> >> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and >> >>> JRockit - and I believe that it is more than enough. Also keep in mind >> >> that >> >>> we already have an alternative Unsafe based memory storage - and >> although >> >>> it's not thoroughly tested for performance it dramaticaly simplifies >> >> code, >> >>> I have great expectations about it. >> >> Me too :-). Yup definitely more simple and faster ! >> >> But we must provide a switch off configuration mechanism if some users >> >> don't want to use that (because Unsafe is just "Unsafe" :-) ) >> >> And sorry I didn't have a look yet at your changes with using Unsafe. >> >>> >> >>> Cheers, >> >>> R >> >>> >> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy >> wrote: >> >>> >> >>>> 2012/9/29 Raffaele P. Guidi : >> >>>>> What do you think about: >> >>>>> 1) implementing a lightning serialization module >> >>>>> 2) creating a serializer that directly works on a directmemory >> >> provider >> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer? >> >>>> >> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as >> >>>> maybe some use other jdks :-) ) >> >>>>> >> >>>>> Now I see lightning is apache licensed and this is fine but I don't >> >> think >> >>>>> it is published in any public maven repo, am I right? We could find a >> >> way >> >>>>> to deal with this; options vary from publishing lightning to the free >> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and >> >> republishing >> >>>>> lightning code in DirectMemory becoming a commiter (which has to >> >> undergo >> >>>> a >> >>>>> PMC vote). >> >>>> >> >>>> At least for the moment he can provide a patch to be integrated in DM >> >> :-) >> >>>> >> >>>>> >> >>>>> I'd like to hear your and the team feelings on this. >> >>>>> >> >>>>> Thanks, >> >>>>> Raffaele >> >>>>> >> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius wrote: >> >>>>> >> >>>>>> Hey Raffaele, >> >>>>>> >> >>>>>> that's quite similar to what I did at work. We're developing Flash >> >>>>>> online games and using a customized AMF serialization. To support >> >>>>>> async writing of the clients event results I added serialization of >> >>>>>> the components / entities to the players zone calculation and stored >> >>>>>> the pre-serialized bytestream directly to the off-heap (using >> >>>>>> direct-ring-cache implementation). When the client requests the >> >>>>>> results (using long-polling) I just write the pre-serialized data to >> >>>>>> the right position to deserialize it by standard ways on Flash side. >> >>>>>> >> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You can >> >>>>>> avoid using byte arrays and minimalize GC. >> >>>>>> >> >>>>>> Cheers >> >>>>>> Chris >> >>>>>> >> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi: >> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a >> >> new >> >>>>>> memory >> >>>>>>> storage implementation using Unsafe (and I did it, recently) and >> >>>> also, as >> >>>>>>> DirectMemory relies heavily on serialization (and supports many of >> >>>> them, >> >>>>>>> protostuff, protobuf, msgpack and of course standard >> >> serialization), >> >>>>>> about >> >>>>>>> creating a simple embedded serializer leveraging the same >> >> techniques >> >>>> you >> >>>>>>> used (Unsafe and bytecode generation). >> >>>>>>> The idea with embedding is avoiding to serialize in a byte array >> >> and >> >>>> then >> >>>>>>> moving the byte array to off-heap memory (via Unsafe or >> >> ByteBuffers), >> >>>> and >> >>>>>>> serializing directly into a "managed" off-heap buffer, thus further >> >>>>>>> optimizing heap utilization (less GC). >> >>>>>>> >> >>>>>>> What do you think about it? Does it make any sense to you? >> >>>>>>> >> >>>>>>> Ciao, >> >>>>>>> R >> >>>>>>> >> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius >> >> wrote: >> >>>>>>> >> >>>>>>>> Hey guys, >> >>>>>>>> >> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few weeks >> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using >> >>>>>>>> Serializer implementation. He told me that he was interested in >> >>>>>>>> adding something similar to DirectMemory and I would be glad to >> >> help >> >>>>>>>> out in this. >> >>>>>>>> >> >>>>>>>> Another project I started a few days ago, since it was needed for >> >>>>>>>> work is DirectRingCache. The name not really meets to actual >> >>>>>>>> implementation since it's not yet a ring buffer using cache. I >> >> used >> >>>>>>>> this for a pre-serialization simple bytestream cache with >> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory having >> >>>>>>>> raw "buffers" to write to or to read from. >> >>>>>>>> >> >>>>>>>> Here are the links from the projects: >> >>>>>>>> https://github.com/noctarius/Lightning >> >>>>>>>> https://github.com/noctarius/direct-ring-cache >> >>>>>>>> >> >>>>>>>> It would be nice to help out since I really like the idea of >> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of >> >> reinventing >> >>>>>>>> the wheel. >> >>>>>>>> >> >>>>>>>> Cheers >> >>>>>>>> Noctarius (Chris) >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Olivier Lamy >> >>>> Talend: http://coders.talend.com >> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy >> >>>> >> >> >> >> >> >> >> >> -- >> >> Olivier Lamy >> >> Talend: http://coders.talend.com >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> >> >> > >> -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy