directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <elecha...@gmail.com>
Subject Re: Ads + Mavibot Partition
Date Mon, 11 Nov 2013 07:14:57 GMT
Le 11/10/13 10:25 AM, Emmanuel Lécharny a écrit :
> Just  quick heads up :
>
> Im tryinh to fix a bug that forbid us to inject a few thousands entries
> in the server (the injection test fails after aound 3000 thousands of
> entries added). The problem lies in teh way we serialize/deserialize
> ParentIdAndRdn. For some reason, we add the length of the byte[]
> containing the ParentIdAndRdn before the serialized value. The
> deserializer does not expect to find this length, thought the next
> position is not good. If I remove this length, now I have a pb because
> the number of expected RDNs is too big, and I get an
> ArrayOutofCoundException while tryingt to create an array of RDN[] with
> zillions of values.
>
> Obviously, we don't serialize/deserialize correctly, although the
> ser/deser methods have been tested and work. It's more about the way we
> call them : when we reach a number of stored elements, the cache is
> full, and we need to read back some pages from disk. This is when we
> have the issue. I have to understand why a Page deserialization fails.
> It should not be such a big issue, but the failure is quite random
> (something I don't understand, though... Not sure that the cache is not
> responsible for this randomness).
>
> I'll keep the list updated.
>
Ok, found the issue.

We have to store the length of a serialized key before the serialized
key, otherwise we have no way to know where to start the deserialization
of the next value. The code that deserialize a page is like :

        // Read each value and key
        for ( int i = 0; i < nbElems; i++ )
        {
            // Read the value (page offsets)
            long offset = OFFSET_SERIALIZER.deserialize( byteBuffer );
            long lastOffset = OFFSET_SERIALIZER.deserialize( byteBuffer );

            ...

            // And the key
            K key = btree.getKeySerializer().deserialize( byteBuffer );
        }

Here, we deserialize the key passing a ByteBuffer, though the position
in this ByteBuffer should be changed after the deserialization.

This is correctly done in the MavibotParentIdAndRdnSerializer :

    @Override
    public ParentIdAndRdn deserialize( ByteBuffer buffer ) throws
IOException
    {
        int len = buffer.getInt();

        ParentIdAndRdn parentIdAndRdn = fromBytes( buffer.array(),
buffer.position() );

        buffer.position( buffer.position() + len );

        return parentIdAndRdn;
    }

Although in the same class, the serialization does not correctly set
this value :

    public byte[] serialize( ParentIdAndRdn parentIdAndRdn )
    {
        try
        {
            int bufferSize = 1024;

            while ( bufferSize < Integer.MAX_VALUE )
            {
                // allocate a big enough buffer for most of the cases
                byte[] buffer = new byte[bufferSize];

                try
                {
                    // The current position. Start at 4, as we will add
the length at pos 0  <<<----------------- Here we are missing something
                    int pos = 0;   
<----------------------------------------------------------------------------
It should be 4

                    ...

                    byte[] result = new byte[pos];
<------------------------------------------------------------- We don't
store the byte[] length
                    System.arraycopy( buffer, 0, result, 0, pos );

                    return result;

So we just have to fix that.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Mime
View raw message