Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of cgreen@conductor.com
 designates 65.55.169.90 as permitted sender)
From: Casey Green <cgreen@conductor.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: A more scalable Kafka to Hadoop InputFormat
Thread-Topic: A more scalable Kafka to Hadoop InputFormat
Thread-Index: AQHP9E5Si22eEvXQtkecfmLYBlhTNA==
Date: Thu, 30 Oct 2014 14:32:25 +0000
Message-ID: <D077C436.49C79%cgreen@conductor.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_D077C43649C79cgreenconductorcom_"
MIME-Version: 1.0

--_000_D077C43649C79cgreenconductorcom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi Folks,

I'm open sourcing a scalable Kafka InputFormat.  As far as I know or am awa=
re of, my version is unique compared to other Kafka InputFormats out there,=
 in that input splits are mapped to Kafka log files, rather than entire Kaf=
ka partitions.  Mapping Kafka log files to input splits scales your Map/Red=
uce job by the amount of data left to consume in a queue, whereas mapping i=
nput splits to entire partitions always gives you a constant number of inpu=
t splits.

I wrote up a blog post about it here<http://www.conductor.com/nightlight/da=
ta-stream-processing-bulk-kafka-hadoop/>, and the source code for my KafkaI=
nputFormat is on github<https://github.com/Conductor/kangaroo>.  Your quest=
ions, comments and feedback are welcomed and much appreciated!

Thanks,
Casey Green


--_000_D077C43649C79cgreenconductorcom_
Content-Type: text/html; charset="iso-8859-1"
Content-ID: <0F3F020A456A3D4A960AB051F7BA187C@namprd02.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami=
ly: Calibri, sans-serif;">
<div>
<div>Hi Folks,</div>
<div><br>
</div>
<div>I&#8217;m open sourcing a scalable Kafka InputFormat. &nbsp;As far as =
I know or am aware of, my version is unique compared to other Kafka InputFo=
rmats out there, in that input splits are mapped to Kafka log files, rather=
 than entire Kafka partitions. &nbsp;Mapping Kafka
 log files to input splits scales your Map/Reduce job by the amount of data=
 left to consume in a queue, whereas mapping input splits to entire partiti=
ons always gives you a constant number of input splits.</div>
<div><br>
</div>
<div>I wrote up a blog post about it&nbsp;<a href=3D"http://www.conductor.c=
om/nightlight/data-stream-processing-bulk-kafka-hadoop/">here</a>, and&nbsp=
;<a href=3D"https://github.com/Conductor/kangaroo">the source code for my K=
afkaInputFormat is on github</a>. &nbsp;Your questions,
 comments and feedback are welcomed and much appreciated!</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Casey Green</div>
</div>
<br>
</body>
</html>

--_000_D077C43649C79cgreenconductorcom_--