ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luke Burton <luke_bur...@icloud.com>
Subject Problems storing RoaringBitmaps
Date Wed, 15 Mar 2017 17:34:24 GMT

Hi there,

I'm storing RoaringBitmaps in Ignite and have encountered an odd serialization issue. Please
forgive the samples below being in Clojure, I've written a small wrapper around most Ignite
APIs that I can use. I think you can catch the gist of what I'm doing, I hope :)

 (let [inst (i/instance "attribute_bitmaps")
        bmp (doto (RoaringBitmap.)
              (.add 9999999))
        srz (fn [it]
              (let [b (ByteArrayOutputStream.)]
                (with-open [o (DataOutputStream. b)]
                  (.serialize it o))
                (.toByteArray b)))]

    (doto inst
      (i/put "test" bmp))

    (byte-streams/print-bytes (srz bmp))
    (byte-streams/print-bytes (srz (i/get inst "test"))))

Here I'm just creating a bitmap and storing it in a cache. I'm then just printing the bytes
of the thing I stored, as well as what it looks like coming back out of Ignite.

I get the following warning in the logs:

10:16:45.730 [djura.local ~ nREPL-worker-3 ~ o.a.i.internal.binary.BinaryContext] Class "org.roaringbitmap.RoaringBitmap"
cannot be serialized using BinaryMarshaller because it either implements Externalizable interface
or have writeObject/readObject methods. OptimizedMarshaller will be used instead and class
instances will be deserialized on the server. Please ensure that all nodes have this class
in classpath. To enable binary serialization either implement Binarylizable interface or set
explicit serializer using BinaryTypeConfiguration.setSerializer() method. 

And really strangely, I get the same number of bytes back but some of them at the end have
been zero'd out (first one is correct, second one went through Ignite):

3A 30 00 00 01 00 00 00  98 00 00 00 10 00 00 00      .0..............
7F 96                                                  ..
3A 30 00 00 01 00 00 00  98 00 00 00 10 00 00 00      .0..............
00 00                                                  ..

The resulting object is still a valid RoaringBitmap, except all the values in the bitmap are

I'm guessing from this and the logs, that OptimizedMarshaller is being used instead of BinaryMarshaller,
and OptimizedMarshaller is not copying the internal fields of the class correctly.

Would the recommended approach here be to create a custom class that extends RoaringBitmap
and implements Binarylizable? I'm not sure if Binarylizable is a suitable approach for this
situation where I don't control the source for the class in question. I have no knowledge
of the internal fields of this class and really just want to ensure it survives the roundtrip
through Ignite by using its own internal serialization mechanism.


View raw message