kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Brugidou <maxime.brugi...@gmail.com>
Subject Shallow iteration and MirrorMaker
Date Wed, 23 Jan 2013 14:39:43 GMT
Hi all,

I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
on 0.7 but it looks similar).

When I enable shallow.iterator.enable=true on the consumer config, (and
disable compression on the producer config, as mentioned in the wiki) the
throughput is really good but the data produced is unreadable since the
messages are taken "as is" by the producer and i just get a bunch of bytes
stored with the ByteBufferMessageSet header and all the info.

I tested using the kafka-console-consumer to read my mirrored data and it's
unreadable (unless I pipe the data to gunzip, which manages to read it).

Did I miss something? Is there anything i can activate on the producer side
of MirrorMaker to send the shallow message "as is"? Did anyone manage to
make MirrorMaker work with shallow.iterator.enable=true?

Thanks for your help,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message