avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Hains <aha...@gmail.com>
Subject Re: Problem with cross-references (avro-maven-plugin)
Date Mon, 04 Aug 2014 19:44:59 GMT
You will probably want to explicitly define the order of the imports. I
have something like this:
         <plugin>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <version>${avro.version}</version>
            <executions>
               <execution>
                  <phase>generate-sources</phase>
                  <goals>
                     <goal>schema</goal>
                  </goals>
                  <configuration>
                     <imports>

<import>${project.basedir}/src/main/avro/imports/MySchema1.avsc</import>

<import>${project.basedir}/src/main/avro/imports/MySchema2.avsc</import>

<import>${project.basedir}/src/main/avro/imports/MySchema3.avsc</import>
                     </imports>

 <sourceDirectory>${project.basedir}/src/main/avro/src</sourceDirectory>

 <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                  </configuration>
               </execution>
            </executions>
         </plugin>

-a


On Mon, Aug 4, 2014 at 12:28 PM, Peterson, Michael <
michael.d.peterson@truvenhealth.com> wrote:

>  Hello,
>
>
>
> We recently refactored our large avro schema into separate schemas to make
> them more composable, since many of them get incorporated into other
> objects.  We were glad to see that the avro-maven-plugin allows this
> composable model rather than having to do one large schema with everything
> embedded.
>
>
>
> However, we have hit a problem – the Parser cannot always resolve the
> cross-references.  I upgraded to avro-1.7.7 (both the core avro and the
> avro-maven-plugin project), but that does not solve the issue.
>
>
>
> The problem is that names/schemas are not always resolved.  Worse, we see
> different behavior on Windows vs. Linux.  Below I show a set up with a
> dummy schema that works on Windows 7, but fails on Linux (tested on Centos
> 6 and Ubuntu 14.04).  We also have a more complicated schema with many
> objects which have a number of cross references (but not circular ones)
> that fails on both Windows and Linux.
>
>
>
> Is this behavior a defect?  Should the Schema.Parser be able to resolve
> these cross-dependencies?
>
>
>
> A larger question is why we need to put things in two directories (top
> level and “imports”).  Couldn’t the Parser be made to resolve references of
> things when they are all in the same directory?
>
>
>
>
>
>
>
> Here is a detailed example – again this one works on Windows but fails on
> Linux:
>
>
>
> u070072@TST imports$ tree
>
> imports
>
> ├── pom.xml   (1.4 KiB)
>
> ├── src
>
> │   ├── main
>
> │      ├── java
>
> │         ├── quux00
>
> │            ├── App.java   (182 B)
>
> │      ├── resources
>
> │         ├── avro
>
> │            ├── Top.avsc   (473 B)
>
> │            ├── import
>
> │               ├── Bottom.avsc   (239 B)
>
> │               ├── Foo.avsc   (307 B)
>
> │               ├── Middle.avsc   (393 B)
>
>
>
>
>
> $ mvn -X generate-sources
>
> [INFO] BUILD FAILURE
>
> [INFO]
> ------------------------------------------------------------------------
>
> [INFO] Total time: 0.786 s
>
> [INFO] Finished at: 2014-08-04T15:06:02-05:00
>
> [INFO] Final Memory: 19M/843M
>
> [INFO]
> ------------------------------------------------------------------------
>
> [ERROR] Failed to execute goal
> org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project
> imports: Execution default of goal
> org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name:
> "Bottom" -> [Help 1]
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project
> imports: Execution default of goal
> org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name:
> "Bottom"
>
>         at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
>
>         at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>
>         at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>
>         at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
>
>         at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
>
>         at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>
>         at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
>
>         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
>
>         at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>
>         at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
>
>         at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
>
>         at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:606)
>
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema failed: *Undefined
> name: "Bottom"*
>
>         at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144)
>
>         at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>
>         ... 19 more
>
> Caused by: org.apache.avro.SchemaParseException: Undefined name: "Bottom"
>
>         at org.apache.avro.Schema.parse(Schema.java:1162)
>
>         at org.apache.avro.Schema.parse(Schema.java:1272)
>
>         at org.apache.avro.Schema.parse(Schema.java:1203)
>
>         at org.apache.avro.Schema$Parser.parse(Schema.java:965)
>
>         at org.apache.avro.Schema$Parser.parse(Schema.java:932)
>
>         at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:73)
>
>         at
> org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:216)
>
>         at
> org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:154)
>
>         at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
>
>         ... 20 more
>
>
>
>
>
> Here are the schema sections:
>
>
>
> $ cat src/main/resources/avro/Top.avsc
>
> {
>
>   "namespace": "quux00.avro",
>
>   "type": "record",
>
>   "name": "Top",
>
>   "fields": [
>
>     {"name": "TopId",     "type": ["null", "string"], "default": null},
>
>     {"name": "TopCode",   "type": ["null", "string"], "default": null},
>
>     {"name": "Middles",   "type": ["null", {"type": "array", "items":
> "Middle"}], "default": null},
>
>     {"name": "Bottom",    "type": ["null", "Bottom"], "default": null},
>
>     {"name": "AFoo",      "type": ["null", "Foo"], "default": null}
>
>   ]
>
> }
>
>
>
> $ cat src/main/resources/avro/import/Middle.avsc
>
> {
>
>   "namespace": "quux00.avro",
>
>   "type": "record",
>
>   "name": "Middle",
>
>   "fields": [
>
>     {"name": "Bottom",      "type": ["null", "Bottom"], "default": null},
>
>     {"name": "MiddleId",    "type": ["null", "string"], "default": null},
>
>     {"name": "MiddleCode",  "type": ["null", "string"], "default": null},
>
>     {"name": "MyFoo",       "type": ["null", "Foo"],    "default": null}
>
>   ]
>
> }
>
>
>
> $ cat src/main/resources/avro/import/Foo.avsc
>
> {
>
>   "namespace": "quux00.avro",
>
>   "type": "record",
>
>   "name": "Foo",
>
>   "fields": [
>
>     {"name": "FooId",      "type": ["null", "string"], "default": null},
>
>     {"name": "FooCode",    "type": ["null", "string"], "default": null},
>
>     {"name": "BottomObj",  "type": ["null", "Bottom"], "default": null}
>
>   ]
>
> }
>
>
>
> $ cat src/main/resources/avro/import/Bottom.avsc
>
> {
>
>   "namespace": "quux00.avro",
>
>   "type": "record",
>
>   "name": "Bottom",
>
>   "fields": [
>
>     {"name": "BottomId",    "type": ["null", "string"], "default": null},
>
>     {"name": "BottomCode",  "type": ["null", "string"], "default": null}
>
>   ]
>
> }
>
>
>
>
>
> And the pom:
>
>
>
> $ cat pom.xml
>
> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance"
>
>          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd">
>
>   <modelVersion>4.0.0</modelVersion>
>
>   <groupId>quux00</groupId>
>
>   <artifactId>imports</artifactId>
>
>   <packaging>jar</packaging>
>
>   <version>1.0-SNAPSHOT</version>
>
>   <name>imports</name>
>
>   <url>http://maven.apache.org</url>
>
>
>
>   <build>
>
>     <plugins>
>
>       <plugin>
>
>         <groupId>org.apache.avro</groupId>
>
>         <artifactId>avro-maven-plugin</artifactId>
>
>         <version>1.7.7</version>
>
>         <executions>
>
>           <execution>
>
>             <phase>generate-sources</phase>
>
>             <goals>
>
>               <goal>schema</goal>
>
>             </goals>
>
>             <configuration>
>
>
> <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
>
>
> <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
>
>               <imports>
>
>                 <import>${basedir}/src/main/resources/avro/import</import>
>
>               </imports>
>
>             </configuration>
>
>           </execution>
>
>         </executions>
>
>       </plugin>
>
>     </plugins>
>
>   </build>
>
>
>
>   <dependencies>
>
>     <dependency>
>
>       <groupId>org.apache.avro</groupId>
>
>       <artifactId>avro</artifactId>
>
>       <version>1.7.7</version>
>
>     </dependency>
>
>   </dependencies>
>
> </project>
>
>
>
>
>
>
>
>
>
> Finally, this doesn’t seem to be an issue with the avro-maven-plugin.  I
> see the same behavior with the avro-tools jar.  It works on Windows:
>
>
>
> $ ls input/
>
> Bottom.avsc  Foo.avsc  Middle.avsc  Top.avsc
>
>
>
> $ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
>
> Input files to compile:
>
>   input\Bottom.avsc
>
>   input\Dog.avsc
>
>   input\Middle.avsc
>
>   input\Top.avsc
>
>
>
> $ tree out/
>
> out
>
> ├── quux00
>
> │   ├── avro
>
> │      ├── Bottom.java   (6.4 KiB)
>
> │      ├── Foo.java   (8.1 KiB)
>
> │      ├── Middle.java   (10.1 KiB)
>
> │      ├── Top.java   (12.0 KiB)
>
>
>
>
>
>
>
> but fails on Linux:
>
>
>
> $ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
>
> Input files to compile:
>
>   input/Foo.avsc
>
>   input/Top.avsc
>
>   input/Middle.avsc
>
>   input/Bottom.avsc
>
> Exception in thread "main" org.apache.avro.SchemaParseException: Undefined
> name: "Bottom"
>
>         at org.apache.avro.Schema.parse(Schema.java:1078)
>
>         at org.apache.avro.Schema.parse(Schema.java:1188)
>
>         at org.apache.avro.Schema.parse(Schema.java:1119)
>
>         at org.apache.avro.Schema$Parser.parse(Schema.java:953)
>
>         at org.apache.avro.Schema$Parser.parse(Schema.java:922)
>
>         at
> org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73)
>
>         at org.apache.avro.tool.Main.run(Main.java:84)
>
>         at org.apache.avro.tool.Main.main(Main.java:73)
>
>
>
>
>
>
>
>
>
>
>
> Thank you,
>
> Michael
>
>
>

Mime
View raw message