kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Werner Daehn (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-4852) ByteBufferSerializer not compatible with offsets
Date Mon, 06 Mar 2017 18:28:33 GMT
Werner Daehn created KAFKA-4852:

             Summary: ByteBufferSerializer not compatible with offsets
                 Key: KAFKA-4852
                 URL: https://issues.apache.org/jira/browse/KAFKA-4852
             Project: Kafka
          Issue Type: Bug
          Components: clients
    Affects Versions:
         Environment: all
            Reporter: Werner Daehn
            Priority: Minor

Quick intro: A ByteBuffer.rewind() resets the position to zero. What if the ByteBuffer was
created with an offset? new ByteBuffer(data, 3, 10)? The ByteBufferSerializer will send from
pos=0 and not from pos=3 onwards.

Solution: No rewind() but flip() for reading a ByteBuffer. That's what the flip is meant for.


Imagine the incoming data comes from a byte[], e.g. a network stream containing topicname,
partition, key, value, ... and you want to create a new ProducerRecord for that. As the constructor
of ProducerRecord requires (topic, partition, key, value) you have to copy from above byte[]
the key and value. That means there is a memcopy taking place. Since the payload can be potentially
large, that introduces a lot of overhead. Twice the memory.

A nice solution to this problem is to simply wrap the network byte[] into new ByteBuffers:
ByteBuffer key = ByteBuffer.wrap(data, keystart, keylength);
ByteBuffer value = ByteBuffer.wrap(data, valuestart, valuelength);
and then use the ByteBufferSerializer instead of the ByteArraySerializer.

But that does not work as the ByteBufferSerializer does a rewind(), hence both, key and value,
will start at position=0 of the data[].

public class ByteBufferSerializer implements Serializer<ByteBuffer> {
    public byte[] serialize(String topic, ByteBuffer data) {

This message was sent by Atlassian JIRA

View raw message