Yep. Avro would be great at that (provided your central consumer is Avro friendly, like a Hadoop system). Make sure that all of your schemas have default values defined for fields so that schema evolution will be easier in the future. On Thu, May 23, 2013 at 4:29 PM, Mark <static.void.dev@gmail.com> wrote: > We're thinking about generating logs and events with Avro and shipping > them to a central collector service via Flume. Is this a valid use case? > > -- Sean
Is anyone aware or knows why that the Ruby RPC example client doesn't work with 1.8.7 (Haven't tried any other versions)? - https://github.com/phunt/avro-rpc-quickstart/tree/master/src/main/ruby I keep receiving the following error: read buffer_length': Socket read 0 bytes. (Avro::IPC::ConnectionClosedException)
We're thinking about generating logs and events with Avro and shipping them to a central collector service via Flume. Is this a valid use case?
Hi Ran,
Unfortunately, there's no real way to manipulate unions in Hive. The Avro
SerDe translates Avro unions into Hive unions correctly, but the support
for accessing those fields is not there. The exception to this is the
[null, T] pattern for nullable fields, which is handled by the Avro SerDe
transparently. This JIRA is tracking imporved union support for Hive, but
it's not being actively worked on:
https://issues.apache.org/jira/browse/HIVE-2390.
Thanks,
Mark
On Thu, May 23, 2013 at 11:45 AM, Scott Carey <scottcarey@apache.org> wrote:
> The Hive mailing list would have more info on the Avro SerDe usage.
>
> In general, a system that does not have union types like Hive (or Pig,
> etc) has to expand a union into multiple fields if there are more than one
> non-null type -- and at most one branch of the union is not null.
>
> For example a record with fields:
>
> {"name":"timestamp", "type":"long", "default":-1}
> {"name":"ipAddress", "type":["IPv4", "IPv6"]}
>
> where IPv4 and IPv6 are previously defined types, would have to expand to
> three fields
> "timestamp", "ipAddress:IPv4", and "ipAddress:IPv6", where only one of
> the last two is not null in any given record.
>
> I do not know what Hive's Avro SerDe does with unions.
>
> On 5/23/13 7:15 AM, "Ran S" <rans@liveperson.com> wrote:
>
> >Hi,
> >We started to work with Avro in CDH4 and to query the Avro files using
> >Hive.
> >This does work fine for us, except for unions.
> >We do not understand how to query the data inside a union using Hive.
> >
> >For example, let's look at the following schema:
> >
> >{
> > "type":"record",
> > "name":"event",
> > "namespace":"com.mysite",
> > "fields":[
> > {
> > "name":"header",
> > "type":{
> > "type":"record", "name":"CommonHeader",
> > "fields":[{ "name":"eventTimeStamp", "type":"long", efault":-1
> >},
> > { "name":"globalUserId", "type":["null", "string"],
> >"default":null } ]
> > },
> > "default":null
> > },
> > {
> > "name":"eventbody",
> > "type":{
> > "type":"record", "name":"eventbody",
> > "fields":[
> > {
> > "name":"body",
> > "type":[
> > "null",
> > {
> > "type":"record",
> > "name":"event1",
> > "fields":[
> > {
> > "name":"event1Header",
> > "type":["null", { "type":"array",
> >"items":"string" }], "default":null
> > },
> > {
> > "name":"event1Body",
> > "type":["null", { "type":"array",
> >"items":"string" }], "default":null
> > }
> > ]
> > },
> > {
> > "type":"record",
> > "name":"event2",
> > "fields":[
> > {
> > "name":"page",
> > "type":{
> > "type":"record", "name":"URL",
> >"fields":[{ "name":"url", "type":"string" }]
> > },
> > "default":null
> > },
> > {
> > "name":"referrer", "type":"string",
> >"default":null
> > }
> > ]
> > }
> > ],
> > "default":null
> > }
> > ]
> > },
> > "default":null
> > }
> >]}
> >
> >Note that "body" is a union of three types:
> >null, "event1" and "event2"
> >
> >So if I want to query fields inside event1, I first need to access it.
> >I then set a HiveQL like this:
> >SELECT eventbody.body.??? from SRC
> >
> >My question is: what shoule I put in the ??? above to make this work?
> >
> >Thank you,
> >Ran
> >
> >
> >
> >--
> >View this message in context:
> >
> http://apache-avro.679487.n3.nabble.com/using-Avro-unions-with-HIVE-tp4027
> >473.html
> >Sent from the Avro - Users mailing list archive at Nabble.com.
>
>
>
Thanks Scott!
So it looks like "fromjson" is mainly meant for processing JSON generated
by "tojson" and not as a general "JSON importing tool" (although it could
be used as such) - it's probably my short attention span, but somehow that
point got lost on me. (As I later learned it also seems that the schema
that the fromjson expects is a simplified version - e.g. specifying a
union will give an error.)
So if I expect to be dealing with data coming in as JSON and would need to
be converting it to Avro - the current "best practice" is to write a
program of your own? This seems like a fairly common thing to do, perhaps
if there isn't a general tool, this could be something useful to hack on
for the Avro project...
Grisha
On Thu, 23 May 2013, Scott Carey wrote:
>
>
> On 5/22/13 2:26 PM, "Gregory (Grisha) Trubetskoy" <grisha@apache.org>
> wrote:
>
>>
>> Hello!
>>
>> I have a test.json file that looks like this:
>>
>> {"first":"John", "last":"Doe", "middle":"C"}
>> {"first":"John", "last":"Doe"}
>>
>> (Second line does NOT have a "middle" element).
>>
>> And I have a test.schema file that looks like this:
>>
>> {"name":"test",
>> "type":"record",
>> "fields": [
>> {"name":"first", "type":"string"},
>> {"name":"middle", "type":"string", "default":""},
>> {"name":"last", "type":"string"}
>> ]}
>>
>> I then try to use fromjson, as follows, and it chokes on the second line:
>>
>> $ java -jar avro-tools-1.7.4.jar fromjson --schema-file test.schema
>> test.json > test.avro
>> Exception in thread "main" org.apache.avro.AvroTypeException: Expected
>> field name not found: middle
>> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
>> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
>> at
>> org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:219)
>> at
>> org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:214)
>> at
>> org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:107
>> )
>> at
>> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.j
>> ava:348)
>> at
>> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.j
>> ava:341)
>> at
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:15
>> 4)
>> at
>> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j
>> ava:177)
>> at
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14
>> 8)
>> at
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13
>> 9)
>> at
>> org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:105)
>> at org.apache.avro.tool.Main.run(Main.java:80)
>> at org.apache.avro.tool.Main.main(Main.java:69)
>>
>>
>> The short story is - I need to convert a bunch of JSON where an element
>> may not be present sometimes, in which case I'd want it to default to
>> something sensible, e.g. blank or null.
>>
>> According to the Schema Resolution "if the reader's record schema has a
>> field that contains a default value, and writer's schema does not have a
>> field with the same name, then the reader should use the default value
>> from its field."
>>
>> I'm clearly missing something obvious, any help would be appreciated!
>
> There are two things that seem to be missing here:
> 1. The fromjson tool is configuring the "writer's schema" (and readers's)
> the one you provided. Avro is expecting every
> JSON fragment you are giving it to have the same schema.
> 2. The tool will not work for all arbitrary json, it expects json in the
> format that the Avro JSON Encoder writes. There are a few differences
> with expectations, primarily when disambiguating union types and maps from
> records.
>
> To perform schema evolution while reading, you may need to separate json
> fragments missing "middle" from those that have it, and run the tool
> twice, with corresponding schemas for each case.
> Alternatively the tool could be modified to handle schema resolution or
> deal with different json encodings as
> well(tools/src/main/java/org/apache/avro/tool/DataFileWriteTool).
>
> Alternatively, you can avoid schema resolution and write two files, one
> with data in each schema after separating the records. Then you can deal
> with schema resolution in a later pass through the data with other tools
> (e.g. data file reader + writer), or only lazily
> when reading resolve the data into the schema you wish.
>
>
>
>>
>> Grisha
>>
>
>
On 5/22/13 2:26 PM, "Gregory (Grisha) Trubetskoy" <grisha@apache.org>
wrote:
>
>Hello!
>
>I have a test.json file that looks like this:
>
>{"first":"John", "last":"Doe", "middle":"C"}
>{"first":"John", "last":"Doe"}
>
>(Second line does NOT have a "middle" element).
>
>And I have a test.schema file that looks like this:
>
>{"name":"test",
> "type":"record",
> "fields": [
> {"name":"first", "type":"string"},
> {"name":"middle", "type":"string", "default":""},
> {"name":"last", "type":"string"}
>]}
>
>I then try to use fromjson, as follows, and it chokes on the second line:
>
>$ java -jar avro-tools-1.7.4.jar fromjson --schema-file test.schema
>test.json > test.avro
>Exception in thread "main" org.apache.avro.AvroTypeException: Expected
>field name not found: middle
> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
> at
>org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:219)
> at
>org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:214)
> at
>org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:107
>)
> at
>org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.j
>ava:348)
> at
>org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.j
>ava:341)
> at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:15
>4)
> at
>org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j
>ava:177)
> at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14
>8)
> at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13
>9)
> at
>org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:105)
> at org.apache.avro.tool.Main.run(Main.java:80)
> at org.apache.avro.tool.Main.main(Main.java:69)
>
>
>The short story is - I need to convert a bunch of JSON where an element
>may not be present sometimes, in which case I'd want it to default to
>something sensible, e.g. blank or null.
>
>According to the Schema Resolution "if the reader's record schema has a
>field that contains a default value, and writer's schema does not have a
>field with the same name, then the reader should use the default value
>from its field."
>
>I'm clearly missing something obvious, any help would be appreciated!
There are two things that seem to be missing here:
1. The fromjson tool is configuring the "writer's schema" (and readers's)
the one you provided. Avro is expecting every
JSON fragment you are giving it to have the same schema.
2. The tool will not work for all arbitrary json, it expects json in the
format that the Avro JSON Encoder writes. There are a few differences
with expectations, primarily when disambiguating union types and maps from
records.
To perform schema evolution while reading, you may need to separate json
fragments missing "middle" from those that have it, and run the tool
twice, with corresponding schemas for each case.
Alternatively the tool could be modified to handle schema resolution or
deal with different json encodings as
well(tools/src/main/java/org/apache/avro/tool/DataFileWriteTool).
Alternatively, you can avoid schema resolution and write two files, one
with data in each schema after separating the records. Then you can deal
with schema resolution in a later pass through the data with other tools
(e.g. data file reader + writer), or only lazily
when reading resolve the data into the schema you wish.
>
>Grisha
>
On 5/22/13 10:34 AM, "Mason" <mason@verbasoftware.com> wrote: >dear list, > >I have what I imagine is a standard setup: a web application generates >data in MySQL, which I want to analyze in Hadoop; I run a nightly >process to extract tables of interest, Avroize, and dump into HDFS. > >This has worked great so far because the tools I'm using make it easy to >load a directory tree of Avros with the same schema. > >The issue is what to do when schema changes occur in the SQL database. I >believe column additions and deletions are handled automatically by the >Avro loaders I'm using, but I need to deal with a column rename. > >My thinking is: I could bake the table schemas at time of ETL into the >Avros, for historical record, but then manually copy that schema out as >a "master" schema and apply it to all Avros for which it's appropriate; >then when a column rename occurs, go back and edit the master schema. > >I've never used an external schema before, so please correct if I >misunderstand how they work. > >Anyone have wisdom to share on this topic? I'd love to hear from anyone >who has done this, or has a better solution. The first thing that comes to mind is the alias feature for field names: http://avro.apache.org/docs/current/spec.html#Aliases If you bare using Avro data files, these contain the schemas at the time of writing for "historical record". The trick is being able to distinguish between someone who renamed a column from "foo" to "fubar" and a case where "foo" was removed and "foobar" added. To do this, one has to have knowledge from the SQL database DDL changes. Once you have this, you can choose your reader schema appropriately -- likely by using the 'latest' schema decorated with field aliases where appropriate, but there are other options. > >-Mason
I think this is simply a feature that has not been added. It appears at
quick glance to be a compatible specification change (it does not break
old code or change the binary format).
Please open a JIRA ticket explaining the use case and we can continue
discussion there.
On 5/23/13 7:51 AM, "Jim Donofrio" <donofrio111@gmail.com> wrote:
>The schema resolution page says:
>
> > if both are enums:
> > if the writer's symbol is not present in the reader's enum, then an
>error is signalled.
>
>Is there a reason you could not allow us to provide one of the symbols
>as a default in the reader so that when the reader read the enum with
>symbol it didnt have, that new symbol would get defaulted to the default
>symbol we provide?
>
>For example with the below schema's this would currently fail if the old
>reader encountered a Enum1.C in the data. Why not provide users the
>option to set "default": "A" for example so that any unknown enum's are
>bined into one? The user might have an unknown enum or something.
>Currently I was going to implement this in my application by just using
>an int type and looking up the ordinal of my own Java Enum. If the int
>is larger than any of the oridinals, the enum would be set to my own
>default value in the enum.
>
>Thanks.
>
>Original schema:
>
>{
> "type": "record",
> "name": "EnumExample",
> "fields": [
> {
> "name": "enum1",
> "type": {
> "type": "enum",
> "name": "Enum1",
> "symbols": ["A", "B"]
> }
> }
> ]
>}
>
>New schema:
>
>{
> "type": "record",
> "name": "EnumExample",
> "fields": [
> {
> "name": "enum1",
> "type": {
> "type": "enum",
> "name": "Enum1",
> "symbols": ["A", "B", "C"]
> }
> }
> ]
>}
>
The Hive mailing list would have more info on the Avro SerDe usage.
In general, a system that does not have union types like Hive (or Pig,
etc) has to expand a union into multiple fields if there are more than one
non-null type -- and at most one branch of the union is not null.
For example a record with fields:
{"name":"timestamp", "type":"long", "default":-1}
{"name":"ipAddress", "type":["IPv4", "IPv6"]}
where IPv4 and IPv6 are previously defined types, would have to expand to
three fields
"timestamp", "ipAddress:IPv4", and "ipAddress:IPv6", where only one of
the last two is not null in any given record.
I do not know what Hive's Avro SerDe does with unions.
On 5/23/13 7:15 AM, "Ran S" <rans@liveperson.com> wrote:
>Hi,
>We started to work with Avro in CDH4 and to query the Avro files using
>Hive.
>This does work fine for us, except for unions.
>We do not understand how to query the data inside a union using Hive.
>
>For example, let's look at the following schema:
>
>{
> "type":"record",
> "name":"event",
> "namespace":"com.mysite",
> "fields":[
> {
> "name":"header",
> "type":{
> "type":"record", "name":"CommonHeader",
> "fields":[{ "name":"eventTimeStamp", "type":"long", efault":-1
>},
> { "name":"globalUserId", "type":["null", "string"],
>"default":null } ]
> },
> "default":null
> },
> {
> "name":"eventbody",
> "type":{
> "type":"record", "name":"eventbody",
> "fields":[
> {
> "name":"body",
> "type":[
> "null",
> {
> "type":"record",
> "name":"event1",
> "fields":[
> {
> "name":"event1Header",
> "type":["null", { "type":"array",
>"items":"string" }], "default":null
> },
> {
> "name":"event1Body",
> "type":["null", { "type":"array",
>"items":"string" }], "default":null
> }
> ]
> },
> {
> "type":"record",
> "name":"event2",
> "fields":[
> {
> "name":"page",
> "type":{
> "type":"record", "name":"URL",
>"fields":[{ "name":"url", "type":"string" }]
> },
> "default":null
> },
> {
> "name":"referrer", "type":"string",
>"default":null
> }
> ]
> }
> ],
> "default":null
> }
> ]
> },
> "default":null
> }
>]}
>
>Note that "body" is a union of three types:
>null, "event1" and "event2"
>
>So if I want to query fields inside event1, I first need to access it.
>I then set a HiveQL like this:
>SELECT eventbody.body.??? from SRC
>
>My question is: what shoule I put in the ??? above to make this work?
>
>Thank you,
>Ran
>
>
>
>--
>View this message in context:
>http://apache-avro.679487.n3.nabble.com/using-Avro-unions-with-HIVE-tp4027
>473.html
>Sent from the Avro - Users mailing list archive at Nabble.com.
For your avro files, double check that snappy is used (use avro-tools to peek at the metadata in the file, or simply view the head in a text editor, the compression codec used will be in the header). Snappy is very fast, most likely the time to read is dominated by deserialization. Avro will be slower than a trivial deserializer (but more compact), but being many times slower is not expected. I am not entirely sure how Hive's Avro serDe works -- it is possible there is a performance issue there. If you were able to get a handful of stack traces (kill -3 or jstack) from the mapper tasks (or a profiler output), it would be very insightful. On 5/23/13 12:42 AM, "nir_zamir" <nir.zamir@gmail.com> wrote: >Hi, > >We're examining the storage of our data in Snappy-compressed files. Since >we >want the data's structure to be self contained, we checked it with Avro >and >with Sequence (both are splittable, which should best utilize our >cluster). > >We tested the performance on a 12GB data (CSV) file, and a 4-nodes cluster >on production environment (very strong machines). > >Compression > >What we did here (for test simplicity) is create two Hive tables: >Avro-based >and Sequence-based. Then we enabled Snappy compression and INSERTed the >data >from the RAW table (consisting of the 12GB file). > >In terms of compression rate, Avro was better: 72% vs. 57%. >In both cases there were 45 mappers, and CPU/Mem were very far from their >limit on all machines. >Since there was no reduce operator, this created 45 files. > >Compression time for Avro took longer: 1.75 minutes vs. 1.2 minutes for >sequence files. > >Decompression > >What we did here was this Hive query: >SELECT COUNT(1) FROM table-name; > >Here was the real difference: it took Avro about *75% longer* to perform >this (3 minutes vs. 0.5 minute). >This was very surprising since for our strong machines the I/O would be >expected to be the bottleneck, and since Avro files are smaller,we >expected >them to be faster to decompress. >The number of mappers in both cases was similar (14 vs. 17) and again, >CPU/Mem didn't seem to be exausted. >Since our most critical time is reading, this issue makes it hard for us >to >be using Avro. > >Maybe we're doing something wrong - your input would be much appreciated! > >Thanks, >Nir > > > >-- >View this message in context: >http://apache-avro.679487.n3.nabble.com/Compressed-Avro-vs-compressed-Sequ >ence-unexpected-results-tp4027467.html >Sent from the Avro - Users mailing list archive at Nabble.com.
The schema resolution page says:
> if both are enums:
> if the writer's symbol is not present in the reader's enum, then an
error is signalled.
Is there a reason you could not allow us to provide one of the symbols
as a default in the reader so that when the reader read the enum with
symbol it didnt have, that new symbol would get defaulted to the default
symbol we provide?
For example with the below schema's this would currently fail if the old
reader encountered a Enum1.C in the data. Why not provide users the
option to set "default": "A" for example so that any unknown enum's are
bined into one? The user might have an unknown enum or something.
Currently I was going to implement this in my application by just using
an int type and looking up the ordinal of my own Java Enum. If the int
is larger than any of the oridinals, the enum would be set to my own
default value in the enum.
Thanks.
Original schema:
{
"type": "record",
"name": "EnumExample",
"fields": [
{
"name": "enum1",
"type": {
"type": "enum",
"name": "Enum1",
"symbols": ["A", "B"]
}
}
]
}
New schema:
{
"type": "record",
"name": "EnumExample",
"fields": [
{
"name": "enum1",
"type": {
"type": "enum",
"name": "Enum1",
"symbols": ["A", "B", "C"]
}
}
]
}
Hi,
We started to work with Avro in CDH4 and to query the Avro files using Hive.
This does work fine for us, except for unions.
We do not understand how to query the data inside a union using Hive.
For example, let's look at the following schema:
{
"type":"record",
"name":"event",
"namespace":"com.mysite",
"fields":[
{
"name":"header",
"type":{
"type":"record", "name":"CommonHeader",
"fields":[{ "name":"eventTimeStamp", "type":"long", efault":-1
},
{ "name":"globalUserId", "type":["null", "string"],
"default":null } ]
},
"default":null
},
{
"name":"eventbody",
"type":{
"type":"record", "name":"eventbody",
"fields":[
{
"name":"body",
"type":[
"null",
{
"type":"record",
"name":"event1",
"fields":[
{
"name":"event1Header",
"type":["null", { "type":"array",
"items":"string" }], "default":null
},
{
"name":"event1Body",
"type":["null", { "type":"array",
"items":"string" }], "default":null
}
]
},
{
"type":"record",
"name":"event2",
"fields":[
{
"name":"page",
"type":{
"type":"record", "name":"URL",
"fields":[{ "name":"url", "type":"string" }]
},
"default":null
},
{
"name":"referrer", "type":"string",
"default":null
}
]
}
],
"default":null
}
]
},
"default":null
}
]}
Note that "body" is a union of three types:
null, "event1" and "event2"
So if I want to query fields inside event1, I first need to access it.
I then set a HiveQL like this:
SELECT eventbody.body.??? from SRC
My question is: what shoule I put in the ??? above to make this work?
Thank you,
Ran
--
View this message in context: http://apache-avro.679487.n3.nabble.com/using-Avro-unions-with-HIVE-tp4027473.html
Sent from the Avro - Users mailing list archive at Nabble.com.
Hi, We're examining the storage of our data in Snappy-compressed files. Since we want the data's structure to be self contained, we checked it with Avro and with Sequence (both are splittable, which should best utilize our cluster). We tested the performance on a 12GB data (CSV) file, and a 4-nodes cluster on production environment (very strong machines). Compression What we did here (for test simplicity) is create two Hive tables: Avro-based and Sequence-based. Then we enabled Snappy compression and INSERTed the data from the RAW table (consisting of the 12GB file). In terms of compression rate, Avro was better: 72% vs. 57%. In both cases there were 45 mappers, and CPU/Mem were very far from their limit on all machines. Since there was no reduce operator, this created 45 files. Compression time for Avro took longer: 1.75 minutes vs. 1.2 minutes for sequence files. Decompression What we did here was this Hive query: SELECT COUNT(1) FROM table-name; Here was the real difference: it took Avro about *75% longer* to perform this (3 minutes vs. 0.5 minute). This was very surprising since for our strong machines the I/O would be expected to be the bottleneck, and since Avro files are smaller,we expected them to be faster to decompress. The number of mappers in both cases was similar (14 vs. 17) and again, CPU/Mem didn't seem to be exausted. Since our most critical time is reading, this issue makes it hard for us to be using Avro. Maybe we're doing something wrong - your input would be much appreciated! Thanks, Nir -- View this message in context: http://apache-avro.679487.n3.nabble.com/Compressed-Avro-vs-compressed-Sequence-unexpected-results-tp4027467.html Sent from the Avro - Users mailing list archive at Nabble.com.
Yes agreed, that would be a good approach to take, however before I go try to figure out how to write a NettyTransceiver in python, I'd like to get some community input on so that: - I understand why there wasn't one built in the first place, and thus answer is it actually possible to build that in python - I'm not duplicating effort if there is somebody already working on it So any NettyServer & NettyTransceiver experts in the house? :) Cheers, Stefan On Wed, May 22, 2013 at 4:22 PM, Atin Sood <soodatin@outlook.com> wrote: > Sure Stefan > > I am a newbie to both python and avro and have limited experience in > networking. > > But if I am not wrong the problem is at python client end as I am guessing > you must be using s.th like > > client code - attach to the server and send a message > > client = ipc.HTTPTransceiver(server_addr[0], server_addr[1]) > requestor = ipc.Requestor(PROTOCOL, client) > > > So even though you can go ahead and switch to NettyServer in java you will be limited to use httpserver because your python client uses http client > > https://github.com/phunt/avro-rpc-quickstart/blob/master/src/main/java/example/Main.java > > I guess the best way to get around this will be to look at source code of avro python client ipc.py file > > and add a new implementation besides the one that comes out of the box. > > class HTTPTransceiver(object): > """ > A simple HTTP-based transceiver implementation. > Useful for clients but not for servers > """ > > > That said, again I am new to the whole thing so I might be totally wrong :) > > > -- > Atin Sood > Sent with Sparrow <http://www.sparrowmailapp.com/?sig> > > On Wednesday, May 22, 2013 at 11:55 AM, Stefan Krawczyk wrote: > > Hi Atin, > > Thanks for the response. Yes I understand I could use HTTPServer on the > java side and things would work. However I'm after a solution where I can > still have the java side use the NettyServer. > > Cheers, > > Stefan > > > On Wed, May 22, 2013 at 4:11 AM, Atin Sood <soodatin@outlook.com> wrote: > > You can try looking into something that I wrote as an example > > > https://github.com/atinsood/HESDataAnalyticsFinalProject/tree/master/javaXPython > > https://github.com/atinsood/HESDataAnalyticsFinalProject#javaxpython > > -- > Atin Sood > Sent with Sparrow <http://www.sparrowmailapp.com/?sig> > > On Tuesday, May 21, 2013 at 11:18 PM, Stefan Krawczyk wrote: > > Hi, > > I am trying to use Avro RPC and have a python client talk to a java > server, using the avro-rpc-quickstart<https://github.com/phunt/avro-rpc-quickstart> on > github as a base (I made sure the avro version being pulled in was 1.7.4). > However when I get my python client to talk to the java server I see this > error: > > 2013-05-20 19:38:32,512 (pool-5-thread-2) [WARN - > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)] > Unexpected exception from downstream. > org.apache.avro.AvroRuntimeException: Excessively large list allocation > request detected: 539959368 items! Connection closed. > at > org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167) > at > org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > > From digging around on the web I understand this is a NettyTransceiver > issue, i.e. the python client isn't using it because it uses the > HTTPTransceiver. > > I was wondering, what are my options for moving forward, other than > getting the java server to use the HTTPTransceiver? > > Apologies if I have overlooked something that points out what I can do. > > Cheers, > > Stefan > > > > >
Sure Stefan
I am a newbie to both python and avro and have limited experience in networking.
But if I am not wrong the problem is at python client end as I am guessing you must be using
s.th like
client code - attach to the server and send a message
client = ipc.HTTPTransceiver(server_addr[0], server_addr[1])
requestor = ipc.Requestor(PROTOCOL, client)
So even though you can go ahead and switch to NettyServer in java you will be limited to use
httpserver because your python client uses http client
https://github.com/phunt/avro-rpc-quickstart/blob/master/src/main/java/example/Main.java
I guess the best way to get around this will be to look at source code of avro python client
ipc.py file
and add a new implementation besides the one that comes out of the box.
class HTTPTransceiver(object): """ A simple HTTP-based transceiver implementation. Useful
for clients but not for servers """
That said, again I am new to the whole thing so I might be totally wrong :)
--
Atin Sood
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Wednesday, May 22, 2013 at 11:55 AM, Stefan Krawczyk wrote:
> Hi Atin,
>
> Thanks for the response. Yes I understand I could use HTTPServer on the java side and
things would work. However I'm after a solution where I can still have the java side use the
NettyServer.
>
> Cheers,
>
> Stefan
>
>
> On Wed, May 22, 2013 at 4:11 AM, Atin Sood <soodatin@outlook.com (mailto:soodatin@outlook.com)>
wrote:
> > You can try looking into something that I wrote as an example
> >
> > https://github.com/atinsood/HESDataAnalyticsFinalProject/tree/master/javaXPython
> >
> > https://github.com/atinsood/HESDataAnalyticsFinalProject#javaxpython
> >
> > --
> > Atin Sood
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Tuesday, May 21, 2013 at 11:18 PM, Stefan Krawczyk wrote:
> >
> > > Hi,
> > >
> > > I am trying to use Avro RPC and have a python client talk to a java server,
using the avro-rpc-quickstart (https://github.com/phunt/avro-rpc-quickstart) on github as
a base (I made sure the avro version being pulled in was 1.7.4). However when I get my python
client to talk to the java server I see this error:
> > >
> > > 2013-05-20 19:38:32,512 (pool-5-thread-2) [WARN - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)]
Unexpected exception from downstream.
> > > org.apache.avro.AvroRuntimeException: Excessively large list allocation request
detected: 539959368 items! Connection closed.
> > > at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167)
> > > at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:722)
> > >
> > > From digging around on the web I understand this is a NettyTransceiver issue,
i.e. the python client isn't using it because it uses the HTTPTransceiver.
> > >
> > > I was wondering, what are my options for moving forward, other than getting
the java server to use the HTTPTransceiver?
> > >
> > > Apologies if I have overlooked something that points out what I can do.
> > >
> > > Cheers,
> > >
> > > Stefan
> >
>
Hello!
I have a test.json file that looks like this:
{"first":"John", "last":"Doe", "middle":"C"}
{"first":"John", "last":"Doe"}
(Second line does NOT have a "middle" element).
And I have a test.schema file that looks like this:
{"name":"test",
"type":"record",
"fields": [
{"name":"first", "type":"string"},
{"name":"middle", "type":"string", "default":""},
{"name":"last", "type":"string"}
]}
I then try to use fromjson, as follows, and it chokes on the second line:
$ java -jar avro-tools-1.7.4.jar fromjson --schema-file test.schema test.json > test.avro
Exception in thread "main" org.apache.avro.AvroTypeException: Expected field name not found:
middle
at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:219)
at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:214)
at org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:107)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:348)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:341)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:154)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:105)
at org.apache.avro.tool.Main.run(Main.java:80)
at org.apache.avro.tool.Main.main(Main.java:69)
The short story is - I need to convert a bunch of JSON where an element
may not be present sometimes, in which case I'd want it to default to
something sensible, e.g. blank or null.
According to the Schema Resolution "if the reader's record schema has a
field that contains a default value, and writer's schema does not have a
field with the same name, then the reader should use the default value
from its field."
I'm clearly missing something obvious, any help would be appreciated!
Grisha
dear list, I have what I imagine is a standard setup: a web application generates data in MySQL, which I want to analyze in Hadoop; I run a nightly process to extract tables of interest, Avroize, and dump into HDFS. This has worked great so far because the tools I'm using make it easy to load a directory tree of Avros with the same schema. The issue is what to do when schema changes occur in the SQL database. I believe column additions and deletions are handled automatically by the Avro loaders I'm using, but I need to deal with a column rename. My thinking is: I could bake the table schemas at time of ETL into the Avros, for historical record, but then manually copy that schema out as a "master" schema and apply it to all Avros for which it's appropriate; then when a column rename occurs, go back and edit the master schema. I've never used an external schema before, so please correct if I misunderstand how they work. Anyone have wisdom to share on this topic? I'd love to hear from anyone who has done this, or has a better solution. -Mason
Hi Atin, Thanks for the response. Yes I understand I could use HTTPServer on the java side and things would work. However I'm after a solution where I can still have the java side use the NettyServer. Cheers, Stefan On Wed, May 22, 2013 at 4:11 AM, Atin Sood <soodatin@outlook.com> wrote: > You can try looking into something that I wrote as an example > > > https://github.com/atinsood/HESDataAnalyticsFinalProject/tree/master/javaXPython > > https://github.com/atinsood/HESDataAnalyticsFinalProject#javaxpython > > -- > Atin Sood > Sent with Sparrow <http://www.sparrowmailapp.com/?sig> > > On Tuesday, May 21, 2013 at 11:18 PM, Stefan Krawczyk wrote: > > Hi, > > I am trying to use Avro RPC and have a python client talk to a java > server, using the avro-rpc-quickstart<https://github.com/phunt/avro-rpc-quickstart> on > github as a base (I made sure the avro version being pulled in was 1.7.4). > However when I get my python client to talk to the java server I see this > error: > > 2013-05-20 19:38:32,512 (pool-5-thread-2) [WARN - > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)] > Unexpected exception from downstream. > org.apache.avro.AvroRuntimeException: Excessively large list allocation > request detected: 539959368 items! Connection closed. > at > org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167) > at > org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > > From digging around on the web I understand this is a NettyTransceiver > issue, i.e. the python client isn't using it because it uses the > HTTPTransceiver. > > I was wondering, what are my options for moving forward, other than > getting the java server to use the HTTPTransceiver? > > Apologies if I have overlooked something that points out what I can do. > > Cheers, > > Stefan > > >
You can try looking into something that I wrote as an example https://github.com/atinsood/HESDataAnalyticsFinalProject/tree/master/javaXPython https://github.com/atinsood/HESDataAnalyticsFinalProject#javaxpython -- Atin Sood Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, May 21, 2013 at 11:18 PM, Stefan Krawczyk wrote: > Hi, > > I am trying to use Avro RPC and have a python client talk to a java server, using the avro-rpc-quickstart (https://github.com/phunt/avro-rpc-quickstart) on github as a base (I made sure the avro version being pulled in was 1.7.4). However when I get my python client to talk to the java server I see this error: > > 2013-05-20 19:38:32,512 (pool-5-thread-2) [WARN - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)] Unexpected exception from downstream. > org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected: 539959368 items! Connection closed. > at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167) > at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139) > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286) > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > > From digging around on the web I understand this is a NettyTransceiver issue, i.e. the python client isn't using it because it uses the HTTPTransceiver. > > I was wondering, what are my options for moving forward, other than getting the java server to use the HTTPTransceiver? > > Apologies if I have overlooked something that points out what I can do. > > Cheers, > > Stefan
Hi, I am trying to use Avro RPC and have a python client talk to a java server, using the avro-rpc-quickstart <https://github.com/phunt/avro-rpc-quickstart> on github as a base (I made sure the avro version being pulled in was 1.7.4). However when I get my python client to talk to the java server I see this error: 2013-05-20 19:38:32,512 (pool-5-thread-2) [WARN - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)] Unexpected exception from downstream. org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected: 539959368 items! Connection closed. at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167) at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) >From digging around on the web I understand this is a NettyTransceiver issue, i.e. the python client isn't using it because it uses the HTTPTransceiver. I was wondering, what are my options for moving forward, other than getting the java server to use the HTTPTransceiver? Apologies if I have overlooked something that points out what I can do. Cheers, Stefan
Waiting for a review : https://issues.apache.org/jira/browse/AVRO-1337 Regards Bertrand On Tue, May 21, 2013 at 9:50 PM, Bertrand Dechoux <dechouxb@gmail.com>wrote: > After browsing the (nice) API, it seems indeed trivial. > > The Idl class allows to read/parse the related file. > > http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#Idl%28java.io.File%29 > > The Protocol object can then be requested from it. > > http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#ProtocolDeclaration%28%29 > > Of course, the types can then be requested from the Protocol itself. > > http://avro.apache.org/docs/current/api/java/org/apache/avro/Protocol.html#getTypes%28%29 > > And then it is only a matter of serializing them. And actually the API > provides even a 'pretty' option. > > http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html#toString%28boolean%29 > > I will definitely look at it (ie with javac). Contributing it as a tool > would be nice. But I won't go into the maven land. > > Regards > > Bertrand > > > On Fri, May 17, 2013 at 9:11 PM, Doug Cutting <cutting@apache.org> wrote: > >> There's not a tool that does this currently. Note however that >> existing tools will generate Java classes for each type in an IDL >> file, so if you're only using Java then you might not need a .avsc >> file for each type in the IDL. >> >> It would not be hard to add a tool (or an option to an existing tool) >> that wrote a .avsc file for each type in an .avdl file. One could >> also add Maven support for this. If this is of interest, please file >> an issue in Jira. >> >> https://issues.apache.org/jira/browse/AVRO >> >> If you're willing and able, please provide an implementation. >> Otherwise hopefully someone else will help out. >> >> Cheers, >> >> Doug >> >> On Fri, May 17, 2013 at 7:10 AM, Jeremy Kahn <trochee@trochee.net> wrote: >> > The "types" field in a protocol (.avro) may get you what you need. The >> > corresponding schema objects should be able to render to well-formed >> avsc >> > objects. >> > >> > On May 17, 2013 5:47 AM, "Bertrand Dechoux" <dechouxb@gmail.com> wrote: >> >> >> >> Hi, >> >> >> >> I have lots of avro schemas and most of them are about complex objects >> (ie >> >> nested definition of stuff). The syntax of avro idl is attractive in >> order >> >> to build something more readable and thus maintainable. However, it >> looks >> >> like I can't generate any avsc from a avdl (or the avpr generated from >> the >> >> avdl). I understand what is a protocol and I don't need one but the idl >> >> syntax is really attractive. Is there really no way to use it for that >> >> purpose? >> >> >> >> Regards >> >> >> >> Bertrand >> >> >> >> >> >> PS : I remember seeing a discussion about that subject but I can't find >> >> it. >> > > > > -- > Bertrand Dechoux > -- Bertrand Dechoux
After browsing the (nice) API, it seems indeed trivial. The Idl class allows to read/parse the related file. http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#Idl%28java.io.File%29 The Protocol object can then be requested from it. http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#ProtocolDeclaration%28%29 Of course, the types can then be requested from the Protocol itself. http://avro.apache.org/docs/current/api/java/org/apache/avro/Protocol.html#getTypes%28%29 And then it is only a matter of serializing them. And actually the API provides even a 'pretty' option. http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html#toString%28boolean%29 I will definitely look at it (ie with javac). Contributing it as a tool would be nice. But I won't go into the maven land. Regards Bertrand On Fri, May 17, 2013 at 9:11 PM, Doug Cutting <cutting@apache.org> wrote: > There's not a tool that does this currently. Note however that > existing tools will generate Java classes for each type in an IDL > file, so if you're only using Java then you might not need a .avsc > file for each type in the IDL. > > It would not be hard to add a tool (or an option to an existing tool) > that wrote a .avsc file for each type in an .avdl file. One could > also add Maven support for this. If this is of interest, please file > an issue in Jira. > > https://issues.apache.org/jira/browse/AVRO > > If you're willing and able, please provide an implementation. > Otherwise hopefully someone else will help out. > > Cheers, > > Doug > > On Fri, May 17, 2013 at 7:10 AM, Jeremy Kahn <trochee@trochee.net> wrote: > > The "types" field in a protocol (.avro) may get you what you need. The > > corresponding schema objects should be able to render to well-formed avsc > > objects. > > > > On May 17, 2013 5:47 AM, "Bertrand Dechoux" <dechouxb@gmail.com> wrote: > >> > >> Hi, > >> > >> I have lots of avro schemas and most of them are about complex objects > (ie > >> nested definition of stuff). The syntax of avro idl is attractive in > order > >> to build something more readable and thus maintainable. However, it > looks > >> like I can't generate any avsc from a avdl (or the avpr generated from > the > >> avdl). I understand what is a protocol and I don't need one but the idl > >> syntax is really attractive. Is there really no way to use it for that > >> purpose? > >> > >> Regards > >> > >> Bertrand > >> > >> > >> PS : I remember seeing a discussion about that subject but I can't find > >> it. > -- Bertrand Dechoux
Hi List, I'm working with the c++ bindings to try to write data to avro files. Much of the documentation assumes that the types to be written (and the code to write the data) are generated using avrogencpp. In my case I have an existing set of type/struct hierarchies that I'm trying to write so I don't want to use the output of avrogencpp directly. Instead I am producing code that is very similar to what avrogencpp outputs but is adapted to suit my types. What I'm finding is that the c++ DataFileWriter does no validation between the schema that I provide and the datums that get written. As such any discrepancy between the schema and the datums that are written causes the file to be corrupted and essentially unreadable. I see that there is a ValidatingEncoder class that can be used when serialising to a memorystream (as per the Getting Started docs) but there doesn't appear to be any method for using this encoder with the DataFileWriter. Am I missing something? Is there a preferred way to make the writer do validation? Thanks Jarrad "Warning: The information contained in this email and any attached files is confidential to BAE Systems Australia. If you are not the intended recipient, any use, disclosure or copying of this email or any attachments is expressly prohibited. If you have received this email in error, please notify us immediately. VIRUS: Every care has been taken to ensure this email and its attachments are virus free, however, any loss or damage incurred in using this email is not the sender's responsibility. It is your responsibility to ensure virus checks are completed before installing any data sent in this email to your computer."
Hi all, I am using a lot of secondary sort Comparators in my avro map reduce jobs. However I haven't found a comfortable way of constructing the grouping schema (sorting schema is mostly the default) or the partitioner for the binary data compare method in AvroKeyComparator. So in my jobs i mostly construct a manual schema with Schema.createRecord() but that is really verbose. I also have some jobs where i created new schema definitions with the correct grouping order but that is tedious either. Does anyone have an idea of how to make this easier? I could think of either some meta annotations in the schema or some property based reflection stuff. In the end it's mostly one field from the record that's partitoned on and one or two that are used as grouping comparators. I think the basic toolset for doing this should be there..I just need some hints.. Cheers, Johannes
There's not a tool that does this currently. Note however that existing tools will generate Java classes for each type in an IDL file, so if you're only using Java then you might not need a .avsc file for each type in the IDL. It would not be hard to add a tool (or an option to an existing tool) that wrote a .avsc file for each type in an .avdl file. One could also add Maven support for this. If this is of interest, please file an issue in Jira. https://issues.apache.org/jira/browse/AVRO If you're willing and able, please provide an implementation. Otherwise hopefully someone else will help out. Cheers, Doug On Fri, May 17, 2013 at 7:10 AM, Jeremy Kahn <trochee@trochee.net> wrote: > The "types" field in a protocol (.avro) may get you what you need. The > corresponding schema objects should be able to render to well-formed avsc > objects. > > On May 17, 2013 5:47 AM, "Bertrand Dechoux" <dechouxb@gmail.com> wrote: >> >> Hi, >> >> I have lots of avro schemas and most of them are about complex objects (ie >> nested definition of stuff). The syntax of avro idl is attractive in order >> to build something more readable and thus maintainable. However, it looks >> like I can't generate any avsc from a avdl (or the avpr generated from the >> avdl). I understand what is a protocol and I don't need one but the idl >> syntax is really attractive. Is there really no way to use it for that >> purpose? >> >> Regards >> >> Bertrand >> >> >> PS : I remember seeing a discussion about that subject but I can't find >> it.
The "types" field in a protocol (.avro) may get you what you need. The corresponding schema objects should be able to render to well-formed avsc objects. On May 17, 2013 5:47 AM, "Bertrand Dechoux" <dechouxb@gmail.com> wrote: > Hi, > > I have lots of avro schemas and most of them are about complex objects (ie > nested definition of stuff). The syntax of avro idl is attractive in order > to build something more readable and thus maintainable. However, it looks > like I can't generate any avsc from a avdl (or the avpr generated from the > avdl). I understand what is a protocol and I don't need one but the idl > syntax is really attractive. Is there really no way to use it for that > purpose? > > Regards > > Bertrand > > > PS : I remember seeing a discussion about that subject but I can't find it. >
Hi, I have lots of avro schemas and most of them are about complex objects (ie nested definition of stuff). The syntax of avro idl is attractive in order to build something more readable and thus maintainable. However, it looks like I can't generate any avsc from a avdl (or the avpr generated from the avdl). I understand what is a protocol and I don't need one but the idl syntax is really attractive. Is there really no way to use it for that purpose? Regards Bertrand PS : I remember seeing a discussion about that subject but I can't find it.
Hi Scott,
How would you envision this working with the avro compiler? Would it be
akin to the capabilites added by
AVRO-1188<https://issues.apache.org/jira/browse/AVRO-1188> where
the existing java enums would be referenced in the importedFiles directory
passed in parameter to the compiler?
Should I file another JIRA for this...?
--
Felix
On Mon, May 13, 2013 at 5:38 PM, Scott Carey <scottcarey@apache.org> wrote:
> It would be nice to be able to reference an existing class when using the
> specific compiler.
>
> If you have an existing "com.mycompany.Foo" enum (or SpecificRecord, or
> Fixed type), then provide the specific compiler with the type prior to
> parsing the schema, it could accept a reference:
>
> {"type":"record", "name":"com.mycompany.Rec", "fields": [
> {"name":"fooField", "type":"com.mycompany.Foo"}
> ]}
>
> Ordinarily, this would fail to compile, but given a reference to an
> existing compatible type, such as an enum, it could work.
>
> -Scott
>
> On 5/9/13 4:39 PM, "Felix GV" <felix@mate1inc.com> wrote:
>
> Hello,
>
> I'm currently writing an avro schema which includes an enum field that I
> already have as a java enum in my application.
>
> At first, I named the avro field with the same fully qualified name
> (package name dot enum name) as my existing java enum. I then ran the avro
> compiler and found that it overwrote my existing java enum with an
> avro-generated enum.
>
> I find this slightly annoying because my java enum had comments
> documenting the purpose of each enum value, and the avro-generated enum
> doesn't have this.
>
> I see two or three potential solutions:
>
> 1. Accepting to replace my current enum with the avro-generated one in
> my code base, which I feel I cannot document properly (since I have access
> to just one doc attribute for the whole enum, instead of per symbol). On a
> side note, I haven't found any way to have a multi-line doc attribute in an
> avro schema, so that makes things slightly more annoying still. I wouldn't
> mind settling on using the avro-generated enums without documentation per
> symbol if at least I could have one big doc/comment that documents all
> symbols at once, but since it seems the doc attribute must be a one-liner,
> this is starting to be a little too messy for my taste...
> 2. Maintaining two separate enums: my manually written (and
> documented) enum as well as the avro-generated enum. For now, I think this
> is what I'm going to do, because those enums have little chances of
> changing anyway, but from a maintenance standpoint, it seems pretty
> horrendous...
> 3. I guess there's a third way, which would involve creating a script
> that backs up my enums, compiles all my schemas, and then restores my
> backed up enums, but this also seems ultra messy :( ... I haven't tested if
> it'd work (since the manually written enum is missing the $SCHEMA field),
> but I guess it would...
>
> Am I being OCD about this? or is this a concern that others have bumped
> into? How do you guys deal with this? Did I miss anything in the way avro
> works?
>
> P.S.: I've seen that reflect mappings may be able to work with arbitrary
> java enums, but since they seemed discouraged for performance reasons, I
> haven't digged much in this direction. I'd like to keep using .avsc files
> if possible, but if there's a better way, I can certainly try it.
>
> P.P.S.: We're currently using avro 1.6.1, but if the latest version
> provides a nice way of handling my use case, then I guess I could get us to
> upgrade...
>
> Thanks a lot :) !
>
> --
> Felix
>
>
FWIW, I have a simple script, avro_to_mongo.pig that does the conversion: https://github.com/rjurney/enron-node-mongo/blob/master/avro_to_mongo.pig On Wed, May 15, 2013 at 4:12 PM, Atin Sood <soodatin@outlook.com> wrote: > I am thinking of a use case of creating a dynamic UI where a user gets to > pick fields that they want and the data type, other attributes like whether > the fi > using this I am thinking of generating an avro schema which can be saved > somewhere > now when the user comes back to the UI which is based out of this schema > and enters a bunch of values I would want to validate this data against > this avro and save it in mongo > > just wondering if anyone has tried saving avro data into mongo > and whats the best way to go about it.. new to avro to there might be a > fundamental flaw in my thinking > -- Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com
I am thinking of a use case of creating a dynamic UI where a user gets to pick fields that they want and the data type, other attributes like whether the fi using this I am thinking of generating an avro schema which can be saved somewhere now when the user comes back to the UI which is based out of this schema and enters a bunch of values I would want to validate this data against this avro and save it in mongo just wondering if anyone has tried saving avro data into mongo and whats the best way to go about it.. new to avro to there might be a fundamental flaw in my thinking
Just trying to validate this is a good approach.We currently have a union
schema that we use to define a simple data item of type "any" (like the old
COM Variant):
{"name": "item", "type": [ "double", "float", "int", "long", "string",
"DateTime", "null"] }
I'd like to add another union member that works like "null", in that no
data ever gets written except the union discriminator itself. We are
streaming time-series data, and this value would have a special meaning of
"value is unchanged". I could make it an enum with just one value, but then
you would write at least two bytes. So I'm thinking I can make a record:
{
"type": "record",
"name": "Unchanged",
"fields":
[
{ "name": "item", "type": "null" }
]
}
and then my union becomes
{
"name": "item", "type": [ "double", "float", "int", "long", "string",
"DateTime", "null", Unchanged]
}
Seem reasonable?
I'm using Avro 1.7.4, and the error is coming between a map step and reduce step. I'm unsure how an Avro record could get corrupted between a mapper and reducer, but it has, how would I be able to handle that exception? On Mon, May 13, 2013 at 3:40 PM, Harsh J <harsh@cloudera.com> wrote: > Its difficult to tell what the error means without context and other > info (such as version). If I had to guess, I think there may be a > corruption on the file being processed here. Does running the file > through avro-tools' tojson sub-command end up in a successful read? > > On Tue, May 14, 2013 at 3:28 AM, Sripad Sriram <sripad@path.com> wrote: > > Hi all, > > > > A java hadoop job that's previously executed without issue began erroring > > with the following stack trace - have any of you seen this before? > > > > java.lang.ArrayIndexOutOfBoundsException: 14 > > at > > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > > at > > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > > at > > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > > at > > > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > > at > > > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > > at > > > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > > at > > > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > > at > > > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:83) > > at > > > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:65) > > at > > org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1262) > > at > > org.apache.hadoop.mapred.Task$ValuesIterator.nextKey(Task.java:1233) > > at > > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:533) > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429) > > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132) > > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > > > -- > Harsh J >
On Wed, May 8, 2013 at 11:49 AM, David Arthur <mumrah@gmail.com> wrote: > I have looked at the Json schema included with Avro, but this requires a > top-level "value" element which I don't want. There's code in Avro that will read and write Jackson JsonNode directly, without creating any intermediate "value" structure. http://avro.apache.org/docs/current/api/java/org/apache/avro/data/Json.html One should be able to easily write a JsonParser and JsonGenerator that read and write directly using this schema, so that Jackson's ObjectCodec could then be used to read and write arbitrary Pojos to Avro data files. Doug
Scott Carey <scottcarey@apache.org> writes:
> Making the DatumReader/Writers configurable would be a welcome
> addition.
Excellent!
> Ideally, much more of what goes on there could be:
> 1. configuration driven
> 2. pre-computed to avoid repeated work during decoding/encoding
>
> We do some of both already. The trick is to do #1 without impacting
> performance and #2 requires a bigger overhaul.
Which work in particular? In my pass through the AvroSerialization
implementation so far, it looks like each MR task would create either
one or two Serializers/Deserializers (key and value), each of which in
turn would create one DatumWriter/DatumReader and Encoder/Decoder pair.
Or do De/Serializers get created multiple times per task?
> If you would like, a contribution including a Clojure related maven
> module or two that depends on the Java stuff would be a welcome
> addition and allow us to identify compatibility issues as we change
> the Java library over time.
That sounds like a great end-goal. Right now at the company I work for
(Damballa) we've just started getting our toes wet with Avro. Avro won
our serialization-format bake-off, but we haven't started actually using
it. I just finished an initial pass at Avro-Clojure integration and we
have released it under an open source license:
https://github.com/damballa/abracad
I would very much like to eventually get a iteration of it into Avro
proper, but I want to actually start using it and Avro first, so we can
hammer out any interface issues etc.
Anyway, I'll try to work up a patch to add some more configuration hooks
to the AvroSerialization. Should I also create a ticket in the Avro
issue tracker?
-Marshall
Its difficult to tell what the error means without context and other info (such as version). If I had to guess, I think there may be a corruption on the file being processed here. Does running the file through avro-tools' tojson sub-command end up in a successful read? On Tue, May 14, 2013 at 3:28 AM, Sripad Sriram <sripad@path.com> wrote: > Hi all, > > A java hadoop job that's previously executed without issue began erroring > with the following stack trace - have any of you seen this before? > > java.lang.ArrayIndexOutOfBoundsException: 14 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:83) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:65) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1262) > at > org.apache.hadoop.mapred.Task$ValuesIterator.nextKey(Task.java:1233) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:533) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132) > at org.apache.hadoop.mapred.Child.main(Child.java:249) -- Harsh J
Making the DatumReader/Writers configurable would be a welcome addition. Ideally, much more of what goes on there could be: 1. configuration driven 2. pre-computed to avoid repeated work during decoding/encoding We do some of both already. The trick is to do #1 without impacting performance and #2 requires a bigger overhaul. If you would like, a contribution including a Clojure related maven module or two that depends on the Java stuff would be a welcome addition and allow us to identify compatibility issues as we change the Java library over time. On 5/8/13 3:33 PM, "Marshall Bockrath-Vandegrift" <llasram@gmail.com> wrote: >Hi all: > >Is there a reason Avro¹s Hadoop serialization classes don¹t allow >configuration of the DatumReader and DatumWriter classes? > >My use-case is that I¹m implementing Clojure DatumReader and -Writer >classes which produce and consume Clojure¹s data structures directly. >I¹d like to then extend that to Hadoop MapReduce jobs which operate in >terms of Clojure data, with Avro handling all de/serialization directly >to/from that Clojure data. > >Am I going around this in a backwards fashion, or would a patch to allow >configuration of the Hadoop serialization DatumReader/Writers be >welcome? > >-Marshall >
The change in the Pig loader in PIG-3297 seems correct they must use CharSequence, not Utf8. I suspect that the Avro 1.5.3.jar does not respect the "avro.java.string" property and is using Utf8 (for the API that Pig is using), but have not confirmed it. "avro.java.string" is an optional hint for the Java implementation. On the Avro side, we may be able to make a modification that allows one to configure a decoder or encoder to ignore the "avro.java.string" property. Perhaps it could look for a system property as an override to help with cases like this. On 5/10/13 3:16 PM, "Michael Moss" <michael.moss@gmail.com> wrote: > Hello, > > It looks like representing avro strings as Utf8 provide some interesting > performance enhancements, but I'm wondering if folks out there are actually > using it in practice, or have had any issues with it. > > We have recently run into an issue where our avro files which represents > strings as "avro.java.string" are causing ClassCastExceptions because Pig and > Hive are expecting them to be Utf8. The exceptions occur when using > avro-1.7.x.jar, but dissapear when using version avro-1.5.3.jar. > > I'm wondering if this is something that should be addressed in the avro jar, > or in pig and hive like this thread suggests: > https://issues.apache.org/jira/browse/PIG-3297 > > Here are the exceptions we are seeing: > Hive: > Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to > org.apache.avro.util.Utf8 at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserializeMap(AvroDeseria > lizer.java:253) > > Pig: > Caused by: java.io.IOException: java.lang.ClassCastException: java.lang.String > cannot be cast to org.apache.avro.util.Utf8 > at > org.apache.pig.piggybank.storage.avro.AvroStorage.getNext(AvroStorage.java:275> ) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.n > extKeyValue(PigRecordReader.java:194) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask. > java:532) > > Thanks. > > -Mike > >
Hi all,
A java hadoop job that's previously executed without issue began erroring
with the following stack trace - have any of you seen this before?
java.lang.ArrayIndexOutOfBoundsException: 14
at
org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
at
org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at
org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at
org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:83)
at
org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:65)
at
org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1262)
at
org.apache.hadoop.mapred.Task$ValuesIterator.nextKey(Task.java:1233)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:533)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
It appears that you will need to modify the JSON decoder in Avro to
achieve this.
The JSON decoder in Avro was built to encode any Avro schema into JSON
with 100% fidelity, so that the decoder can read it back. The decoder
does not work with any arbitrary JSON.
This is because there are ambiguities:
In your example:
{
"id": "doc1",
"fields": {
"foo": "bar",
"spam": "eggs",
"answer": 42,
"x": {"a": 1}
}
}
This can be interpreted by Avro in several ways. Is the value of "fields"
a map or a record with four fields? is the value of "x" a map or a record
with one field? Is "answer" an int, long, float, or double? is a string
"doc1" a string or a bytes literal?
If you want to bake in the assumption that it is "maps, all the way down",
you'll need to extend / modify the JSON Decoder.
It would be a useful contribution to have a generic JSON schema and
decoder for it. We could have a "JSON" schema record (one field, a union
of null, string, double, and map of string to self) and this type's field
would automatically be un-nested by the special JSON decoder and not
interpreted as a record.
-Scott
On 5/8/13 11:49 AM, "David Arthur" <mumrah@gmail.com> wrote:
>I'm attempting to use Jackson and Avro together to map JSON documents to
>a generated Avro class. I have looked at the Json schema included with
>Avro, but this requires a top-level "value" element which I don't want.
>Essentially, I have JSON documents that have a few typed top level
>fields, and one field called "fields" which is more or less arbitrary
>JSON.
>
>I've reduced this down to strings and ints for simplicity
>
>My first attempt was:
>
> {
> "type": "record",
> "name": "Json",
> "fields": [
> {
> "name": "value",
> "type": [ "string", "int", {"type": "map", "values": "Json"} ]
> }
> ]
> },
>
> {
> "name": "Document",
> "type": "record",
> "fields": [
> {
> "name": "id",
> "type": "string"
> },
> {
> "name": "fields",
> "type": {"type": "map", "values": ["string", "int", {"type":
>"map", "values": "Json"}]}
> }
> ]
> }
>
>Given a JSON document like:
>
>{
> "id": "doc1",
> "fields": {
> "foo": "bar",
> "spam": "eggs",
> "answer": 42,
> "x": {"a": 1}
> }
>}
>
>this seems to work, but it doesn't. When I turn around and try to
>serialize this object with Avro, I get the following exception:
>
>java.lang.ClassCastException: java.lang.Integer cannot be cast to
>org.apache.avro.generic.IndexedRecord
> at org.apache.avro.generic.GenericData.getField(GenericData.java:526)
> at org.apache.avro.generic.GenericData.getField(GenericData.java:541)
> at
>org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.
>java:104)
> at
>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:6
>6)
> at
>org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.jav
>a:173)
> at
>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:6
>9)
> at
>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:7
>3)
> at
>org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.jav
>a:173)
> at
>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:6
>9)
> at
>org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.
>java:106)
> at
>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:6
>6)
> at
>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:5
>8)
>
>My best guess is that since the "fields" field is a union, the
>representation of it in the generate class is an Object which Jackson
>happily throws whatever into.
>
>If I change my schema to explicitly use "int" instead of the "Json"
>type, it works fine for my test document
>
> "type": {"type": "map", "values": ["string", "int", {"type":
>"map", "values": "int"}]}
>
>However now I need to enumerate the types for each level of nesting I
>want. This is not ideal, and limits me to a fixed level of nesting
>
>To be clear, my issue is not modelling my schema in Avro, but rather
>getting Jackson to map JSON onto the generated classes without too much
>pain. I have also tried
>https://github.com/FasterXML/jackson-dataformat-avro without much luck.
>
>Any help is appreciated
>
>-David
>
>
>
>
>
It would be nice to be able to reference an existing class when using the
specific compiler.
If you have an existing "com.mycompany.Foo" enum (or SpecificRecord, or
Fixed type), then provide the specific compiler with the type prior to
parsing the schema, it could accept a reference:
{"type":"record", "name":"com.mycompany.Rec", "fields": [
{"name":"fooField", "type":"com.mycompany.Foo"}
]}
Ordinarily, this would fail to compile, but given a reference to an existing
compatible type, such as an enum, it could work.
-Scott
On 5/9/13 4:39 PM, "Felix GV" <felix@mate1inc.com> wrote:
> Hello,
>
> I'm currently writing an avro schema which includes an enum field that I
> already have as a java enum in my application.
>
> At first, I named the avro field with the same fully qualified name (package
> name dot enum name) as my existing java enum. I then ran the avro compiler and
> found that it overwrote my existing java enum with an avro-generated enum.
>
> I find this slightly annoying because my java enum had comments documenting
> the purpose of each enum value, and the avro-generated enum doesn't have this.
>
> I see two or three potential solutions:
> 1. Accepting to replace my current enum with the avro-generated one in my code
> base, which I feel I cannot document properly (since I have access to just one
> doc attribute for the whole enum, instead of per symbol). On a side note, I
> haven't found any way to have a multi-line doc attribute in an avro schema, so
> that makes things slightly more annoying still. I wouldn't mind settling on
> using the avro-generated enums without documentation per symbol if at least I
> could have one big doc/comment that documents all symbols at once, but since
> it seems the doc attribute must be a one-liner, this is starting to be a
> little too messy for my taste...
> 2. Maintaining two separate enums: my manually written (and documented) enum
> as well as the avro-generated enum. For now, I think this is what I'm going to
> do, because those enums have little chances of changing anyway, but from a
> maintenance standpoint, it seems pretty horrendous...
> 3. I guess there's a third way, which would involve creating a script that
> backs up my enums, compiles all my schemas, and then restores my backed up
> enums, but this also seems ultra messy :( ... I haven't tested if it'd work
> (since the manually written enum is missing the $SCHEMA field), but I guess it
> would...
> Am I being OCD about this? or is this a concern that others have bumped into?
> How do you guys deal with this? Did I miss anything in the way avro works?
>
> P.S.: I've seen that reflect mappings may be able to work with arbitrary java
> enums, but since they seemed discouraged for performance reasons, I haven't
> digged much in this direction. I'd like to keep using .avsc files if possible,
> but if there's a better way, I can certainly try it.
>
> P.P.S.: We're currently using avro 1.6.1, but if the latest version provides a
> nice way of handling my use case, then I guess I could get us to upgrade...
>
> Thanks a lot :) !
>
> --
> Felix