avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Pivarski (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AVRO-1422) JSON-deserialization of recursively defined record causes stack overflow
Date Sat, 28 Dec 2013 00:03:50 GMT
Jim Pivarski created AVRO-1422:
----------------------------------

             Summary: JSON-deserialization of recursively defined record causes stack overflow
                 Key: AVRO-1422
                 URL: https://issues.apache.org/jira/browse/AVRO-1422
             Project: Avro
          Issue Type: Bug
          Components: java
    Affects Versions: 1.7.5
         Environment: Linux (but it doesn't matter because it's Java).
            Reporter: Jim Pivarski


A schema defined like this:

{code:title=badSchema.avsc|borderStyle=solid}
{"type": "record",
 "name": "RecursiveRecord",
 "fields": [
   {"name": "child", "type": "RecursiveRecord"}
 ]}
{code}

results in an infinite loop/stack overflow when ingesting JSON that looks like {{{"child":
null}}} or {{{"child": {"null": null}}}}.  For instance, I can compile and load the schema
into a Scala REPL and then cause the error when trying to read in the JSON, like this:

{code:title=command-line-1|borderStyle=solid}
java -jar avro-tools-1.7.5.jar compile schema recursiveSchema.avsc .
javac RecursiveRecord.java -cp avro-tools-1.7.5.jar
scala -cp avro-tools-1.7.5.jar:.
{code}
{code:title=scala-repl-specific-1|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.specific.SpecificDatumReader;

var output: RecursiveRecord = new RecursiveRecord();
val schema: Schema = output.getSchema();
val reader: SpecificDatumReader[RecursiveRecord] = new SpecificDatumReader[RecursiveRecord](schema);
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null":
null}}"""));
{code}

The same is true if I attempt to load it into a generic object:

{code:title=scala-repl-generic-1|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericDatumReader;

val parser = new Schema.Parser();
val schema: Schema = parser.parse("""{"type": "record", "name": "RecursiveRecord", "fields":
[{"name": "child", "type": "RecursiveRecord"}]}""");
val reader: GenericDatumReader[java.lang.Object] = new GenericDatumReader[java.lang.Object](schema);
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null":
null}}"""));
{code}

In all cases, it is the {{reader.read}} calls that cause stack overflows (all four of the
ones described above).  The stack trace is apparently truncated, but what is shown repeats
these two lines until cut off by the JVM:
{code:title=stack-trace|borderStyle=solid}
        at org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:324)
        at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:217)
{code}

The same is not true if we (correctly?) declare the child as a union of null and a recursive
record.  For instance,

{code:title=goodSchema.avsc|borderStyle=solid}
{"type": "record",
 "name": "RecursiveRecord2",
 "fields": [
   {"name": "child", "type": ["RecursiveRecord2", "null"]}
 ]}
{code}
{code:title=command-line-2|borderStyle=solid}
java -jar avro-tools-1.7.5.jar compile schema recursiveSchema2.avsc .
javac RecursiveRecord2.java -cp avro-tools-1.7.5.jar
scala -cp avro-tools-1.7.5.jar:.
{code}
{code:title=scala-repl-specific-2|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.specific.SpecificDatumReader;

var output: RecursiveRecord2 = new RecursiveRecord2();
val schema: Schema = output.getSchema();
val reader: SpecificDatumReader[RecursiveRecord2] = new SpecificDatumReader[RecursiveRecord2](schema);
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null":
null}}"""));
{code}
{code:title=scala-repl-generic-2|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericDatumReader;

val parser = new Schema.Parser()
val schema: Schema = parser.parse("""{"type": "record", "name": "RecursiveRecord2", "fields":
[{"name": "child", "type": ["RecursiveRecord2", "null"]}]}""");
val reader: GenericDatumReader[java.lang.Object] = new GenericDatumReader[java.lang.Object](schema);
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null":
null}}"""));
{code}

For both specific and generic, {{RecursiveRecord2}} works properly: it produces an object
with recursive type and {{child == null}}.

My understanding of the official schema is that only {{RecursiveRecord2}} should be allowed
to have a null {{child}}, so the JSON I supplied would not have been valid input for {{RecursiveRecord}}.
 (If so, then it wouldn't even be possible to give it valid finite input.)  However, it should
give a different error than a stack overflow, something to tell me that {{{"child": null}}}
is not legal unless field {{child}} is declared as a union that includes {{null}}.

The reason one might want this (recursively defined types) is to make trees.  The example
I gave had only one child for simplicity (i.e. it was a linked list), but the error would
apply to binary trees as well.  For instance, here's a three-node list (a little cumbersome
in JSON):

{code:title=motivating-example|borderStyle=solid}
{"child": {"RecursiveRecord2": {"child": {"RecursiveRecord2": {"child": null}}}}}
{code}

I haven't tested this in Avro deserialization (which would be a more reasonable use-case),
but I don't know of a way to generate the Avro-encoded data without first getting it from
human-typable JSON.  (I'm not constructing the Avro byte stream by hand.)




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message