avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AVRO-983) maven-avro-plugin: Allow maven schema compiler to support external reference in an another avro schema file
Date Mon, 29 Oct 2012 17:56:13 GMT

     [ https://issues.apache.org/jira/browse/AVRO-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated AVRO-983:
---------------------------

    Attachment: AVRO-983.patch

Here's a simpler patch that follows Doug's suggestion of reusing Schema.Parser. I haven't
written a unit test but I did verify it manually by writing a schema that referenced another
schema and checked that without the patch the plugin failed and with the patch it successfully
compiled all the files.

There is however a big caveat: cross references only work if the type is defined in a file
that is compiled before it is referenced.

So if a.avsc defines type A which depends on type B defined in b.avsc then compilation will
fail. That's because when a.avsc is compiled first (since it is lexically before b.avsc) B
will not have been read.

It's possible to rename avsc files to make this work, so this patch is an improvement; but
it would be better to improve this in the future to either

* give more control over the order in which the files are compiled, like the command line
version. At the moment they are specified as directories in the Maven POM, so the order is
the lexical ordering of the files in the directory.
* or enhance the compiler to support type references between files. This is more involved
and would require a two pass approach.
                
> maven-avro-plugin: Allow maven schema compiler to support external reference in an another
avro schema file
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: AVRO-983
>                 URL: https://issues.apache.org/jira/browse/AVRO-983
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.6.1
>            Reporter: C├ędric Torcq
>         Attachments: AVRO-983.patch, SchemaMojo.java.patch
>
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
>
> Avro Maven plugin doesn't support an external reference in an another Avro schema file
even this schema is available in the same src directory
> Example :
> {code:javascript} 
> [
> 	{
> 		"namespace": "com.afklm.karma.demand.inter",
> 		"name": "DatedFlightLegInterType",
> 		"type": "record",
> 		"fields":
> 			[
> 				{ "name": "legOrgArp", "type": "string" },
> 				{ "name": "legDestArp", "type": "string" },
> 				{ "name": "legDepDate", "type": "long" },
> 				{ "name": "flightNb", "type": "string" },
> 				{ "name": "dfl", "type": "com.afklm.karma.demand.output.DatedFlightLegType" }
> 			]
> 	},
> 	{
> 		"namespace": "com.afklm.karma.demand.inter",
> 		"name": "AirportAirportDatedFlightLegInterType",
> 		"type": "record",
> 		"fields":
> 			[
> 				{ "name": "aa", "type": "AirportAirportInterType" },
> 				{ "name": "legOrgArp", "type": "string" },
> 				{ "name": "legDestArp", "type": "string" },
> 				{ "name": "legDepDate", "type": "long" },
> 				{ "name": "flightNb", "type": "string" }
> 			]
> 	}
> ]
> {code} 
> generate a fatal error :
> {code} 
> org.apache.avro.SchemaParseException: "com.afklm.karma.demand.output.DatedFlightLegType"
is not a defined name. The type of t
> he "dfl" field must be a defined name or a {"type": ...} expression.
>         at org.apache.avro.Schema.parse(Schema.java:1131)
>         at org.apache.avro.Schema.parse(Schema.java:1200)
>         at org.apache.avro.Schema$Parser.parse(Schema.java:968)
>         at org.apache.avro.Schema$Parser.parse(Schema.java:945)
>         at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:58)
>         at org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:137)
>         at org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:107)
> {code}
> I propose a patch to support also this case :
> In org.apache.avro.mojo.SchemaMojo (line 55):
> 		// File src = new File(sourceDirectory, filename);
> 		// Schema.Parser parser = new Schema.Parser();
> 		// Schema schema = parser.parse(src);
> 		// SpecificCompiler compiler = new SpecificCompiler(schema);
> 		// compiler.setTemplateDir(templateDirectory);
> 		// compiler.setStringType(StringType.valueOf(stringType));
> 		// compiler.compileSchema(src,
> 		// outputDirectory);
> 		
> 		
>         DirectoryScanner scanner = new DirectoryScanner();
>         scanner.setBasedir(sourceDirectory);
>         scanner.setIncludes( getIncludes() );
>         scanner.addDefaultExcludes();
>         scanner.scan();
>         File[] avroFiles = new File[scanner.getIncludedFiles().length];
>         for (int i = 0; i < avroFiles.length; i++) {
>         	avroFiles[i] = new File( scanner.getBasedir(), scanner.getIncludedFiles()[i]
);
> 		}
>         SpecificCompiler.compileSchema(avroFiles, outputDirectory);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message