kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Brugidou <maxime.brugi...@gmail.com>
Subject Re: Shallow iteration and MirrorMaker
Date Wed, 23 Jan 2013 17:30:23 GMT
MirrorMaker between two 0.8 clusters.


On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Hi Maxime,
>
> Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
> data between a Kafka 0.7 and Kafka 0.8 cluster ?
> We have fixed the KafkaMigrationTool to always turn off shallow iteration
> since we have to decompress data in 0.7 format before sending it to a 0.8
> broker.
>
> Thanks,
> Neha
>
>
> On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> <maxime.brugidou@gmail.com>wrote:
>
> > Hi all,
> >
> > I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
> > on 0.7 but it looks similar).
> >
> > When I enable shallow.iterator.enable=true on the consumer config, (and
> > disable compression on the producer config, as mentioned in the wiki) the
> > throughput is really good but the data produced is unreadable since the
> > messages are taken "as is" by the producer and i just get a bunch of
> bytes
> > stored with the ByteBufferMessageSet header and all the info.
> >
> > I tested using the kafka-console-consumer to read my mirrored data and
> it's
> > unreadable (unless I pipe the data to gunzip, which manages to read it).
> >
> > Did I miss something? Is there anything i can activate on the producer
> side
> > of MirrorMaker to send the shallow message "as is"? Did anyone manage to
> > make MirrorMaker work with shallow.iterator.enable=true?
> >
> > Thanks for your help,
> > Maxime
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message