directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <elecha...@gmail.com>
Subject Re: Ads + Mavibot Partition
Date Mon, 11 Nov 2013 08:57:16 GMT
Ok, pb fixed.

It was slightly more complex that I expected. The key length wasn't
stored for nodes, as shown in my previous mail, but the impact was a bit
more than just what I thought (ie, the fix was not what I suggested).


Le 11/11/13 8:14 AM, Emmanuel Lécharny a écrit :
> Le 11/10/13 10:25 AM, Emmanuel Lécharny a écrit :
>> Just  quick heads up :
>>
>> Im tryinh to fix a bug that forbid us to inject a few thousands entries
>> in the server (the injection test fails after aound 3000 thousands of
>> entries added). The problem lies in teh way we serialize/deserialize
>> ParentIdAndRdn. For some reason, we add the length of the byte[]
>> containing the ParentIdAndRdn before the serialized value. The
>> deserializer does not expect to find this length, thought the next
>> position is not good. If I remove this length, now I have a pb because
>> the number of expected RDNs is too big, and I get an
>> ArrayOutofCoundException while tryingt to create an array of RDN[] with
>> zillions of values.
>>
>> Obviously, we don't serialize/deserialize correctly, although the
>> ser/deser methods have been tested and work. It's more about the way we
>> call them : when we reach a number of stored elements, the cache is
>> full, and we need to read back some pages from disk. This is when we
>> have the issue. I have to understand why a Page deserialization fails.
>> It should not be such a big issue, but the failure is quite random
>> (something I don't understand, though... Not sure that the cache is not
>> responsible for this randomness).
>>
>> I'll keep the list updated.
>>
> Ok, found the issue.
>
> We have to store the length of a serialized key before the serialized
> key, otherwise we have no way to know where to start the deserialization
> of the next value. The code that deserialize a page is like :
>
>         // Read each value and key
>         for ( int i = 0; i < nbElems; i++ )
>         {
>             // Read the value (page offsets)
>             long offset = OFFSET_SERIALIZER.deserialize( byteBuffer );
>             long lastOffset = OFFSET_SERIALIZER.deserialize( byteBuffer );
>
>             ...
>
>             // And the key
>             K key = btree.getKeySerializer().deserialize( byteBuffer );
>         }
>
> Here, we deserialize the key passing a ByteBuffer, though the position
> in this ByteBuffer should be changed after the deserialization.
>
> This is correctly done in the MavibotParentIdAndRdnSerializer :
>
>     @Override
>     public ParentIdAndRdn deserialize( ByteBuffer buffer ) throws
> IOException
>     {
>         int len = buffer.getInt();
>
>         ParentIdAndRdn parentIdAndRdn = fromBytes( buffer.array(),
> buffer.position() );
>
>         buffer.position( buffer.position() + len );
>
>         return parentIdAndRdn;
>     }
>
> Although in the same class, the serialization does not correctly set
> this value :
>
>     public byte[] serialize( ParentIdAndRdn parentIdAndRdn )
>     {
>         try
>         {
>             int bufferSize = 1024;
>
>             while ( bufferSize < Integer.MAX_VALUE )
>             {
>                 // allocate a big enough buffer for most of the cases
>                 byte[] buffer = new byte[bufferSize];
>
>                 try
>                 {
>                     // The current position. Start at 4, as we will add
> the length at pos 0  <<<----------------- Here we are missing something
>                     int pos = 0;   
> <----------------------------------------------------------------------------
> It should be 4
>
>                     ...
>
>                     byte[] result = new byte[pos];
> <------------------------------------------------------------- We don't
> store the byte[] length
>                     System.arraycopy( buffer, 0, result, 0, pos );
>
>                     return result;
>
> So we just have to fix that.
>


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Mime
View raw message