Mailing-List: contact avro-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: avro-user@hadoop.apache.org
Message-ID: <4BDF01C8.6060901@apache.org>
Date: Mon, 03 May 2010 10:03:04 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: avro-user@hadoop.apache.org
Subject: Re: references to other schemas
References: <x2w83290b461005021218nafe0e11fha8db2d6fd178ae52@mail.gmail.com>
 <93A14209-033A-4369-8E7D-6B3D585D23C1@richrelevance.com>
In-Reply-To: <93A14209-033A-4369-8E7D-6B3D585D23C1@richrelevance.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Scott Carey wrote:
> There has been talk that AvroGen would handle features like this (as well as many others) in time.  However this is one that should probably be addressed at the JSON level regardless of the future direction of AvroGen.

Note that JSON schemas and protocols need to be standalone, containing 
the full lexical closure of schemas referenced, when they are included 
in data files and exchanged in RPC handshakes without reference to 
external data.  Thus I am reluctant to add a JSON syntax for file 
inclusion.  Rather, I think a pre-processor is appropriate.  The 
pre-processor would not be run on schemas included in files or exchanged 
in RPC handshakes, but would be run for schemas read from files.

I have experimented with using the m4 pre-processor for this purpose, 
and found it a bit awkward.  Perhaps someone can develop macros for m4 
that make it palatable, or perhaps we can develop a custom pre-processor 
for JSON.

We might exploit otherwise-illegal JSON syntax, like backquotes, for 
pre-processor directives.  An include might look something like:

{"protocol": "org.foo.BarProtocol",
  "types": [
    `include org.foo.Bar`,
     ...
   ]
}

Also note that a protocol file (.avpr) need not actually define any 
messages but can be used to define a set of types that reference one 
another.  This is a stopgap, but a useful one.

Doug