Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: pass (nike.apache.org: domain of ewasserman@247-inc.com
 designates 216.32.181.181 as permitted sender)
Received-SPF: pass (mail4-ch1: domain of 247-inc.com designates 111.221.112.37
 as permitted sender) client-ip=111.221.112.37;
 envelope-from=ewasserman@247-inc.com;
 helo=HKNPRD0310HT002.apcprd03.prod.outlook.com ;.outlook.com ;
From: Eric Wasserman <ewasserman@247-inc.com>
To: Mark <static.void.dev@gmail.com>
CC: "user@avro.apache.org" <user@avro.apache.org>
Subject: Re: Schema Registry?
Thread-Topic: Schema Registry?
Thread-Index: AQHOncEmq5LydoG6o0SUD0rv+ql3L5meYzAAgAHKSoCAAEiKZg==
Date: Thu, 22 Aug 2013 01:41:47 +0000
Message-ID: <975796BD-D717-4A65-98CB-D96E77111C71@247-inc.com>
References: <140DC851-A8C4-4FAA-8325-7685646CF881@gmail.com>
 <6D4574B3-4C4D-4A24-AE18-E18E0DBAA89F@247-inc.com>,<006414B3-8152-4443-B1F1-263B9A4A4E46@gmail.com>
In-Reply-To: <006414B3-8152-4443-B1F1-263B9A4A4E46@gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Yes we have a Kafka event consumer that creates the files in HDFS. There ar=
e other non-Hadoop consumers as well.=20

On Aug 21, 2013, at 2:23 PM, "Mark" <static.void.dev@gmail.com> wrote:

> Some final questions.
>=20
> Since there is no need for the schema in each Kafka event do you just out=
put the message without the container file (file header, metadata, sync_mar=
kers)? If so, how do you get this working with the Kafka hadoop consumers? =
Doing it this way, does it require you to write your own consumer to write =
to hadoop?
>=20
> Thanks
>=20
> On Aug 20, 2013, at 11:01 AM, Eric Wasserman <ewasserman@247-inc.com> wro=
te:
>=20
>> You may want to check out this Avro feature request: https://issues.apac=
he.org/jira/browse/AVRO-1124
>> which has a lot of nice motivation and usage patterns. Unfortunately, it=
s not yet a resolved request.
>>=20
>> There are really two broad use cases.=20
>>=20
>> 1) The data are "small" compared to the schema (perhaps because its a co=
llection or stream of records encoded by different schemas)
>> 2) The data are "big" compared to the schema. (very big records or lots =
of records that share a schema)
>>=20
>> Case (1) is often a candidate for a schema registry. Case (2) not as muc=
h.
>>=20
>> Examples from my own usage:
>>=20
>> For Kafka we include an MD5 digest of the writer's schema with each Mess=
age. It is serialized as a concatenation of the fixed-length MD5 and the bi=
nary Avro-encoded data. To decode we read off the MD5, look up the schema a=
nd use it to decode the remainder of the Message.
>> [You could also segregate data written with different schemas into diffe=
rent Kafka topics. By knowing which topic a message is under you then arran=
ge a way to look up the writer's schema. That lets you avoid even the cost =
of including the MD5 in the Messages.]
>>=20
>> In either case consumer code needs to look up the full schema from a "re=
gistry" in order to do the actual decode the Avro-encoded data. The registr=
y serves the full schema that corresponds to the specified MD5 digest.
>>=20
>> We use a similar technique for storing MD5-tagged Avro data in "columns"=
 of Cassandra and so on.
>>=20
>> Case (2) is pretty well handled by just embedding the full schema itself=
.
>>=20
>> For example, for Hadoop you can just use Avro data files which include t=
he actual schema in a header. All the record in the file then adhere to tha=
t same schema. In this case using a registry to get the writer's schema is =
not necessary.
>>=20
>> Note: As described in the feature request linked above, some people use =
a schema registry as a way of coordinating schema evolution rather than jus=
t as a way of making schema access "economical".
>>=20
>>=20
>>=20
>> On Aug 20, 2013, at 9:19 AM, Mark wrote:
>>=20
>>> Can someone break down how message serialization would work with Avro a=
nd a schema registry? We are planning to use Avro with Kafka and I've read =
instead of adding a schema to every single event it would be wise to add so=
me sort of fingerprint with each message to identify which schema it should=
 used. What I'm having trouble understanding is, how do we read the fingerp=
rint without a schema? Don't we need the schema to deserialize?  Same quest=
ion goes for working with Hadoop.. how does the input format know which sch=
ema to use?
>>>=20
>>> Thanks
>>=20
>>=20
>=20
>=20