Mailing-List: contact users-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@kafka.apache.org
Received-SPF: pass (nike.apache.org: domain of maxime.brugidou@gmail.com
 designates 209.85.223.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOG_4QbTQ=-tdaL-w7pf45UswOFg9Arqy9nCXY6rr2-NJw6D8Q@mail.gmail.com>
References: 
 <CAHReGahfHteMDvySyfXSz+LV9hsNOTJjAZZ3CwCXxgkTseA6Gg@mail.gmail.com>
	<CAOG_4QYYun4m4qsrwb270=kZCFbyafrWtHtHUDd5XELc2tH6eA@mail.gmail.com>
	<CAHReGago9d+6HOwbqmXkbz4OR0WtX-etR9DvS3c0-dpmVx1+7A@mail.gmail.com>
	<CAOG_4QbTQ=-tdaL-w7pf45UswOFg9Arqy9nCXY6rr2-NJw6D8Q@mail.gmail.com>
Date: Thu, 24 Jan 2013 09:55:16 +0100
Message-ID: 
 <CAHReGahn=e4EKXNYESswvJXycMMErFKovsv0hPBfCLHotfV4Lw@mail.gmail.com>
Subject: Re: Shallow iteration and MirrorMaker
From: Maxime Brugidou <maxime.brugidou@gmail.com>
To: users@kafka.apache.org
Content-Type: multipart/alternative; boundary=e89a8f3b9c1f41c00e04d404f81b

--e89a8f3b9c1f41c00e04d404f81b
Content-Type: text/plain; charset=ISO-8859-1

Thanks for your response, here is the JIRA:
https://issues.apache.org/jira/browse/KAFKA-732


On Thu, Jan 24, 2013 at 1:16 AM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> We haven't tried to run this. Please can you file a JIRA ?
>
>
> On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
> <maxime.brugidou@gmail.com>wrote:
>
> > MirrorMaker between two 0.8 clusters.
> >
> >
> > On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Hi Maxime,
> > >
> > > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to
> mirror
> > > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > > We have fixed the KafkaMigrationTool to always turn off shallow
> iteration
> > > since we have to decompress data in 0.7 format before sending it to a
> 0.8
> > > broker.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > > <maxime.brugidou@gmail.com>wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am working with MirrorMaker on the 0.8 branch (so I haven't tested
> > this
> > > > on 0.7 but it looks similar).
> > > >
> > > > When I enable shallow.iterator.enable=true on the consumer config,
> (and
> > > > disable compression on the producer config, as mentioned in the wiki)
> > the
> > > > throughput is really good but the data produced is unreadable since
> the
> > > > messages are taken "as is" by the producer and i just get a bunch of
> > > bytes
> > > > stored with the ByteBufferMessageSet header and all the info.
> > > >
> > > > I tested using the kafka-console-consumer to read my mirrored data
> and
> > > it's
> > > > unreadable (unless I pipe the data to gunzip, which manages to read
> > it).
> > > >
> > > > Did I miss something? Is there anything i can activate on the
> producer
> > > side
> > > > of MirrorMaker to send the shallow message "as is"? Did anyone manage
> > to
> > > > make MirrorMaker work with shallow.iterator.enable=true?
> > > >
> > > > Thanks for your help,
> > > > Maxime
> > > >
> > >
> >
>

--e89a8f3b9c1f41c00e04d404f81b--