Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: pass (athena.apache.org: domain of russell.jurney@gmail.com
 designates 209.85.220.171 as permitted sender)
References: <445E44A6-345F-42FC-B340-48AF3804A93D@braindump.ms>
 <CALEq1Z8s1sfaAVB7YE2rpZ=v3q1V_h7Vm39h0HsOzxJ+qfQRSg@mail.gmail.com>
 <65FA8140-FBFF-4EEA-A607-F00620DE2AAD@braindump.ms>
 <CALEq1Z-dSxBuAYpsBnOpHXZFPv1p6fKvMxYG=XCdRoOqhiK82g@mail.gmail.com>
From: Russell Jurney <russell.jurney@gmail.com>
In-Reply-To: 
 <CALEq1Z-dSxBuAYpsBnOpHXZFPv1p6fKvMxYG=XCdRoOqhiK82g@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Date: Tue, 18 Sep 2012 16:18:30 -0700
Message-ID: <-2429953882573608962@unknownmsgid>
Subject: Re: Converting arbitrary JSON to avro
To: "user@avro.apache.org" <user@avro.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Fwiw, I do this in web apps all the time via the python avro lib and json.d=
umps

Russell Jurney
twitter.com/rjurney
russell.jurney@gmail.com
datasyndrome.com

On Sep 18, 2012, at 12:38 PM, Doug Cutting <cutting@apache.org> wrote:

> On Tue, Sep 18, 2012 at 11:34 AM, Markus Strickler <markus@braindump.ms> =
wrote:
>> Json.Writer is indeed what I had in mind and I have successfully managed=
 to convert my existing JSON to avro using it.
>> However using GenericDatumReader on this feels pretty unnatural, as I se=
em to be unable to access fields directly. It seems I have to access the "v=
alue" field on each record which returns a Map which uses Utf8 Objects as k=
eys for the actual fields. Or am I doing something wrong here?
>
> Hmm.  We could re-factor Json.SCHEMA so the union is the top-level
> element.  That would get rid of the wrapper around every value.  It's
> a more redundant way to write the schema, but the binary encoding is
> identical (since a record wrapper adds no bytes).  It would hence
> require no changes to Json.Reader or Json.Writer.
>
> [ "long",
>  "double",
>  "string",
>  "boolean",
>  "null",
>  {"type" : "array",
>   "items" : {
>       "type" : "record",
>       "name" : "org.apache.avro.data.Json",
>       "fields" : [ {
>           "name" : "value",
>           "type" : [ "long", "double", "string", "boolean", "null",
>                      {"type" : "array", "items" : "Json"},
>                      {"type" : "map", "values" : "Json"}
>                    ]
>       } ]
>   }
>  },
>  {"type" : "map", "values" : "Json"}
> ]
>
> You can try this by placing this schema in
> share/schemas/org/apache/avro/data/Json.avsc and re-building the avro
> jar.
>
> Would such a change be useful to you?  If so, please file an issue in Jir=
a.
>
> Or we could even refactor this schema so that a Json object is the
> top-level structure:
>
> {"type" : "map",
> "values" : [ "long",
>              "double",
>              "string",
>              "boolean",
>              "null",
>              {"type" : "array",
>               "items" : {
>                   "type" : "record",
>                   "name" : "org.apache.avro.data.Json",
>                   "fields" : [ {
>                       "name" : "value",
>                       "type" : [ "long", "double", "string", "boolean", "=
null",
>                                  {"type" : "array", "items" : "Json"},
>                                  {"type" : "map", "values" : "Json"}
>                                ]
>                   } ]
>               }
>              },
>              {"type" : "map", "values" : "Json"}
>            ]
> }
>
> This would change the binary format but would not change the
> representation that GenericDatumReader would hand you from my first
> example above (since the generic representation unwraps unions).
> Using this schema would require changes to Json.Writer and
> Json.Reader.  It would better conform to the definition of Json, which
> only permits objects as the top-level type.
>
>> Concerning the more specific schema, you are of course completely right.=
 Unfortunately more or less all the fields in the JSON data format are opti=
onal and many have substructures, so, at least in my understanding, I have =
to use unions of null and the actual type throughout the schema. I tried us=
ing JsonDecoder first (or rather the fromjson option of the avro tool, whic=
h, I think, uses JsonDecoder) but given the current JSON structures, this d=
idn't work.
>
>> So I'll probably have to look into implementing my own converter.  Howev=
er given the rather complex structure of the original JSON I'm wondering if=
 trying to represent the data in avro is such a good idea in the first plac=
e.
>
> It would be interesting to see whether, with the appropriate schema,
> whether the dataset is smaller and faster to process as Avro than as
> Json.  If you have 1000 fields in your data but the typical record
> only has one or two non-null, then an Avro record is perhaps not a
> good representation.  An Avro map might be better, but if the values
> are similarly variable then Json might be competitive.
>
> Cheers,
>
> Doug