avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peterson, Michael" <michael.d.peter...@truvenhealth.com>
Subject RE: Problem with cross-references (avro-maven-plugin)
Date Mon, 04 Aug 2014 20:58:54 GMT
Yes, thanks Adrian.  That solved it for both my simple example and the more complex production
schema.

Still this seems pretty fragile.  Shouldn’t the Parser be set up to resolve these (ideally
without an “import” directory)?  Should there be a feature enhancement request for this?

In any case, thanks for solving the immediate issue.

-Michael

From: Adrian Hains [mailto:ahains@gmail.com]
Sent: Monday, August 04, 2014 3:45 PM
To: user@avro.apache.org
Subject: Re: Problem with cross-references (avro-maven-plugin)

You will probably want to explicitly define the order of the imports. I have something like
this:
         <plugin>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <version>${avro.version}</version>
            <executions>
               <execution>
                  <phase>generate-sources</phase>
                  <goals>
                     <goal>schema</goal>
                  </goals>
                  <configuration>
                     <imports>
                        <import>${project.basedir}/src/main/avro/imports/MySchema1.avsc</import>
                        <import>${project.basedir}/src/main/avro/imports/MySchema2.avsc</import>
                        <import>${project.basedir}/src/main/avro/imports/MySchema3.avsc</import>
                     </imports>
                     <sourceDirectory>${project.basedir}/src/main/avro/src</sourceDirectory>
                     <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                  </configuration>
               </execution>
            </executions>
         </plugin>

-a

On Mon, Aug 4, 2014 at 12:28 PM, Peterson, Michael <michael.d.peterson@truvenhealth.com<mailto:michael.d.peterson@truvenhealth.com>>
wrote:
Hello,

We recently refactored our large avro schema into separate schemas to make them more composable,
since many of them get incorporated into other objects.  We were glad to see that the avro-maven-plugin
allows this composable model rather than having to do one large schema with everything embedded.

However, we have hit a problem – the Parser cannot always resolve the cross-references.
 I upgraded to avro-1.7.7 (both the core avro and the avro-maven-plugin project), but that
does not solve the issue.

The problem is that names/schemas are not always resolved.  Worse, we see different behavior
on Windows vs. Linux.  Below I show a set up with a dummy schema that works on Windows 7,
but fails on Linux (tested on Centos 6 and Ubuntu 14.04).  We also have a more complicated
schema with many objects which have a number of cross references (but not circular ones) that
fails on both Windows and Linux.

Is this behavior a defect?  Should the Schema.Parser be able to resolve these cross-dependencies?

A larger question is why we need to put things in two directories (top level and “imports”).
 Couldn’t the Parser be made to resolve references of things when they are all in the same
directory?



Here is a detailed example – again this one works on Windows but fails on Linux:

u070072@TST imports$ tree
imports
├── pom.xml   (1.4 KiB)
├── src
│   ├── main
│      ├── java
│         ├── quux00
│            ├── App.java   (182 B)
│      ├── resources
│         ├── avro
│            ├── Top.avsc   (473 B)
│            ├── import
│               ├── Bottom.avsc   (239 B)
│               ├── Foo.avsc   (307 B)
│               ├── Middle.avsc   (393 B)


$ mvn -X generate-sources
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.786 s
[INFO] Finished at: 2014-08-04T15:06:02-05:00
[INFO] Final Memory: 19M/843M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on
project imports: Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom" -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.avro:avro-maven-plugin:1.7.7:schema
(default) on project imports: Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom"
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom"
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 19 more
Caused by: org.apache.avro.SchemaParseException: Undefined name: "Bottom"
        at org.apache.avro.Schema.parse(Schema.java:1162)
        at org.apache.avro.Schema.parse(Schema.java:1272)
        at org.apache.avro.Schema.parse(Schema.java:1203)
        at org.apache.avro.Schema$Parser.parse(Schema.java:965)
        at org.apache.avro.Schema$Parser.parse(Schema.java:932)
        at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:73)
        at org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:216)
        at org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:154)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
        ... 20 more


Here are the schema sections:

$ cat src/main/resources/avro/Top.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Top",
  "fields": [
    {"name": "TopId",     "type": ["null", "string"], "default": null},
    {"name": "TopCode",   "type": ["null", "string"], "default": null},
    {"name": "Middles",   "type": ["null", {"type": "array", "items": "Middle"}], "default":
null},
    {"name": "Bottom",    "type": ["null", "Bottom"], "default": null},
    {"name": "AFoo",      "type": ["null", "Foo"], "default": null}
  ]
}

$ cat src/main/resources/avro/import/Middle.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Middle",
  "fields": [
    {"name": "Bottom",      "type": ["null", "Bottom"], "default": null},
    {"name": "MiddleId",    "type": ["null", "string"], "default": null},
    {"name": "MiddleCode",  "type": ["null", "string"], "default": null},
    {"name": "MyFoo",       "type": ["null", "Foo"],    "default": null}
  ]
}

$ cat src/main/resources/avro/import/Foo.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Foo",
  "fields": [
    {"name": "FooId",      "type": ["null", "string"], "default": null},
    {"name": "FooCode",    "type": ["null", "string"], "default": null},
    {"name": "BottomObj",  "type": ["null", "Bottom"], "default": null}
  ]
}

$ cat src/main/resources/avro/import/Bottom.avsc
{
  "namespace": "quux00.avro",
  "type": "record",
  "name": "Bottom",
  "fields": [
    {"name": "BottomId",    "type": ["null", "string"], "default": null},
    {"name": "BottomCode",  "type": ["null", "string"], "default": null}
  ]
}


And the pom:

$ cat pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>quux00</groupId>
  <artifactId>imports</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>imports</name>
  <url>http://maven.apache.org</url>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.7.7</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>schema</goal>
            </goals>
            <configuration>
              <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
              <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
              <imports>
                <import>${basedir}/src/main/resources/avro/import</import>
              </imports>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <dependencies>
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.7.7</version>
    </dependency>
  </dependencies>
</project>




Finally, this doesn’t seem to be an issue with the avro-maven-plugin.  I see the same behavior
with the avro-tools jar.  It works on Windows:

$ ls input/
Bottom.avsc  Foo.avsc  Middle.avsc  Top.avsc

$ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
Input files to compile:
  input\Bottom.avsc
  input\Dog.avsc
  input\Middle.avsc
  input\Top.avsc

$ tree out/
out
├── quux00
│   ├── avro
│      ├── Bottom.java   (6.4 KiB)
│      ├── Foo.java   (8.1 KiB)
│      ├── Middle.java   (10.1 KiB)
│      ├── Top.java   (12.0 KiB)



but fails on Linux:

$ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
Input files to compile:
  input/Foo.avsc
  input/Top.avsc
  input/Middle.avsc
  input/Bottom.avsc
Exception in thread "main" org.apache.avro.SchemaParseException: Undefined name: "Bottom"
        at org.apache.avro.Schema.parse(Schema.java:1078)
        at org.apache.avro.Schema.parse(Schema.java:1188)
        at org.apache.avro.Schema.parse(Schema.java:1119)
        at org.apache.avro.Schema$Parser.parse(Schema.java:953)
        at org.apache.avro.Schema$Parser.parse(Schema.java:922)
        at org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73)
        at org.apache.avro.tool.Main.run(Main.java:84)
        at org.apache.avro.tool.Main.main(Main.java:73)





Thank you,
Michael


Mime
View raw message