directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng, Kai" <kai.zh...@intel.com>
Subject RE: A default backend based on Google FlatBuffers?
Date Sun, 20 Sep 2015 12:22:49 GMT
>> What I don't get is how you go from a Java instance to a flatbuffer...
Simply saying, it just copies the byte representation of each field of the object to the flat
buffer, as what we did in ApacheDS 'by hand'. The different might be that, the 'by hand' codes
are generated from a predefined schema file, instead of manually composing. No Java serialization
is involved. Note Java is just one of the listed supported languages. 

For now one thing I'm not sure about is how large could the flat buffer that's supported.
It may not work if the content is too large to be held in memory without extra paging support
like we did in Mavibot/Btree I guess. 

Regards,
Kai

-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Sunday, September 20, 2015 12:40 AM
To: Apache Directory Developers List <dev@directory.apache.org>
Subject: Re: A default backend based on Google FlatBuffers?

Le 19/09/15 13:55, Zheng, Kai a écrit :
> Thanks Emmanuel for the feedback.
>
> Yeah, it needs encoding, that means the flatbuffers format should be used to store entries.
>
> It does avoid decoding data, because it operates directly on the binary data. In details,
one may mmap loads the content from a file and gets a bytebuffer. Then all the entries can
be looked up directly on the bytebuffer. As you said one may find an entry efficiently, I
thought a mapping from key to the start address of the corresponding object would be required.
Given the start address of the object, then all the fields of the object can be directly retrieved
without any decoding.

What I don't get is how you go from a Java instance to a flatbuffer...
Because, make no mistake, the costly processing is the serialization. In flatbuffer, how is
this serialization done ?

For instance, in ApacheDS, we serialize entries and other data using our own implementation,
not depending on any Java default serialization (through reflection). the gain is massive.
That's what I don't get :
what's the flatbuffers offer that is better than what we do when serializing 'by hand' ?


Mime
View raw message