avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peterson, Michael" <michael.d.peter...@truvenhealth.com>
Subject Problem with cross-references (avro-maven-plugin)
Date Mon, 04 Aug 2014 19:28:24 GMT
Hello,

We recently refactored our large avro schema into separate schemas to make them more composable,
since many of them get incorporated into other objects.  We were glad to see that the avro-maven-plugin
allows this composable model rather than having to do one large schema with everything embedded.

However, we have hit a problem - the Parser cannot always resolve the cross-references.  I
upgraded to avro-1.7.7 (both the core avro and the avro-maven-plugin project), but that does
not solve the issue.

The problem is that names/schemas are not always resolved.  Worse, we see different behavior
on Windows vs. Linux.  Below I show a set up with a dummy schema that works on Windows 7,
but fails on Linux (tested on Centos 6 and Ubuntu 14.04).  We also have a more complicated
schema with many objects which have a number of cross references (but not circular ones) that
fails on both Windows and Linux.

Is this behavior a defect?  Should the Schema.Parser be able to resolve these cross-dependencies?

A larger question is why we need to put things in two directories (top level and "imports").
 Couldn't the Parser be made to resolve references of things when they are all in the same
directory?



Here is a detailed example - again this one works on Windows but fails on Linux:

u070072@TST imports$ tree
imports
├── pom.xml   (1.4 KiB)
├── src
│   ├── main
│      ├── java
│         ├── quux00
│            ├── App.java   (182 B)
│      ├── resources
│         ├── avro
│            ├── Top.avsc   (473 B)
│            ├── import
│               ├── Bottom.avsc   (239 B)
│               ├── Foo.avsc   (307 B)
│               ├── Middle.avsc   (393 B)


$ mvn -X generate-sources
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.786 s
[INFO] Finished at: 2014-08-04T15:06:02-05:00
[INFO] Final Memory: 19M/843M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on
project imports: Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom" -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.avro:avro-maven-plugin:1.7.7:schema
(default) on project imports: Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom"
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom"
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 19 more
Caused by: org.apache.avro.SchemaParseException: Undefined name: "Bottom"
        at org.apache.avro.Schema.parse(Schema.java:1162)
        at org.apache.avro.Schema.parse(Schema.java:1272)
        at org.apache.avro.Schema.parse(Schema.java:1203)
        at org.apache.avro.Schema$Parser.parse(Schema.java:965)
        at org.apache.avro.Schema$Parser.parse(Schema.java:932)
        at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:73)
        at org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:216)
        at org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:154)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
        ... 20 more


Here are the schema sections:

$ cat src/main/resources/avro/Top.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Top",
  "fields": [
    {"name": "TopId",     "type": ["null", "string"], "default": null},
    {"name": "TopCode",   "type": ["null", "string"], "default": null},
    {"name": "Middles",   "type": ["null", {"type": "array", "items": "Middle"}], "default":
null},
    {"name": "Bottom",    "type": ["null", "Bottom"], "default": null},
    {"name": "AFoo",      "type": ["null", "Foo"], "default": null}
  ]
}

$ cat src/main/resources/avro/import/Middle.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Middle",
  "fields": [
    {"name": "Bottom",      "type": ["null", "Bottom"], "default": null},
    {"name": "MiddleId",    "type": ["null", "string"], "default": null},
    {"name": "MiddleCode",  "type": ["null", "string"], "default": null},
    {"name": "MyFoo",       "type": ["null", "Foo"],    "default": null}
  ]
}

$ cat src/main/resources/avro/import/Foo.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Foo",
  "fields": [
    {"name": "FooId",      "type": ["null", "string"], "default": null},
    {"name": "FooCode",    "type": ["null", "string"], "default": null},
    {"name": "BottomObj",  "type": ["null", "Bottom"], "default": null}
  ]
}

$ cat src/main/resources/avro/import/Bottom.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Bottom",
  "fields": [
    {"name": "BottomId",    "type": ["null", "string"], "default": null},
    {"name": "BottomCode",  "type": ["null", "string"], "default": null}
  ]
}


And the pom:

$ cat pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>quux00</groupId>
  <artifactId>imports</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>imports</name>
  <url>http://maven.apache.org</url>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.7.7</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>schema</goal>
            </goals>
            <configuration>
              <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
              <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
              <imports>
                <import>${basedir}/src/main/resources/avro/import</import>
              </imports>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <dependencies>
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.7.7</version>
    </dependency>
  </dependencies>
</project>




Finally, this doesn't seem to be an issue with the avro-maven-plugin.  I see the same behavior
with the avro-tools jar.  It works on Windows:

$ ls input/
Bottom.avsc  Foo.avsc  Middle.avsc  Top.avsc

$ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
Input files to compile:
  input\Bottom.avsc
  input\Dog.avsc
  input\Middle.avsc
  input\Top.avsc

$ tree out/
out
├── quux00
│   ├── avro
│      ├── Bottom.java   (6.4 KiB)
│      ├── Foo.java   (8.1 KiB)
│      ├── Middle.java   (10.1 KiB)
│      ├── Top.java   (12.0 KiB)



but fails on Linux:

$ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
Input files to compile:
  input/Foo.avsc
  input/Top.avsc
  input/Middle.avsc
  input/Bottom.avsc
Exception in thread "main" org.apache.avro.SchemaParseException: Undefined name: "Bottom"
        at org.apache.avro.Schema.parse(Schema.java:1078)
        at org.apache.avro.Schema.parse(Schema.java:1188)
        at org.apache.avro.Schema.parse(Schema.java:1119)
        at org.apache.avro.Schema$Parser.parse(Schema.java:953)
        at org.apache.avro.Schema$Parser.parse(Schema.java:922)
        at org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73)
        at org.apache.avro.tool.Main.run(Main.java:84)
        at org.apache.avro.tool.Main.main(Main.java:73)





Thank you,
Michael


Mime
View raw message