Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@avro.apache.org
Date: Sat, 28 Dec 2013 00:03:50 +0000 (UTC)
From: "Jim Pivarski (JIRA)" <jira@apache.org>
To: dev@avro.apache.org
Message-ID: <JIRA.12686586.1388188963792.17216.1388189030131@arcas>
In-Reply-To: <JIRA.12686586.1388188963792@arcas>
References: <JIRA.12686586.1388188963792@arcas>
Subject: [jira] [Created] (AVRO-1422) JSON-deserialization of recursively
 defined record causes stack overflow
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Jim Pivarski created AVRO-1422:
----------------------------------

             Summary: JSON-deserialization of recursively defined record causes stack overflow
                 Key: AVRO-1422
                 URL: https://issues.apache.org/jira/browse/AVRO-1422
             Project: Avro
          Issue Type: Bug
          Components: java
    Affects Versions: 1.7.5
         Environment: Linux (but it doesn't matter because it's Java).
            Reporter: Jim Pivarski


A schema defined like this:

{code:title=badSchema.avsc|borderStyle=solid}
{"type": "record",
 "name": "RecursiveRecord",
 "fields": [
   {"name": "child", "type": "RecursiveRecord"}
 ]}
{code}

results in an infinite loop/stack overflow when ingesting JSON that looks like {{{"child": null}}} or {{{"child": {"null": null}}}}.  For instance, I can compile and load the schema into a Scala REPL and then cause the error when trying to read in the JSON, like this:

{code:title=command-line-1|borderStyle=solid}
java -jar avro-tools-1.7.5.jar compile schema recursiveSchema.avsc .
javac RecursiveRecord.java -cp avro-tools-1.7.5.jar
scala -cp avro-tools-1.7.5.jar:.
{code}
{code:title=scala-repl-specific-1|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.specific.SpecificDatumReader;

var output: RecursiveRecord = new RecursiveRecord();
val schema: Schema = output.getSchema();
val reader: SpecificDatumReader[RecursiveRecord] = new SpecificDatumReader[RecursiveRecord](schema);
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null": null}}"""));
{code}

The same is true if I attempt to load it into a generic object:

{code:title=scala-repl-generic-1|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericDatumReader;

val parser = new Schema.Parser();
val schema: Schema = parser.parse("""{"type": "record", "name": "RecursiveRecord", "fields": [{"name": "child", "type": "RecursiveRecord"}]}""");
val reader: GenericDatumReader[java.lang.Object] = new GenericDatumReader[java.lang.Object](schema);
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null": null}}"""));
{code}

In all cases, it is the {{reader.read}} calls that cause stack overflows (all four of the ones described above).  The stack trace is apparently truncated, but what is shown repeats these two lines until cut off by the JVM:
{code:title=stack-trace|borderStyle=solid}
        at org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:324)
        at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:217)
{code}

The same is not true if we (correctly?) declare the child as a union of null and a recursive record.  For instance,

{code:title=goodSchema.avsc|borderStyle=solid}
{"type": "record",
 "name": "RecursiveRecord2",
 "fields": [
   {"name": "child", "type": ["RecursiveRecord2", "null"]}
 ]}
{code}
{code:title=command-line-2|borderStyle=solid}
java -jar avro-tools-1.7.5.jar compile schema recursiveSchema2.avsc .
javac RecursiveRecord2.java -cp avro-tools-1.7.5.jar
scala -cp avro-tools-1.7.5.jar:.
{code}
{code:title=scala-repl-specific-2|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.specific.SpecificDatumReader;

var output: RecursiveRecord2 = new RecursiveRecord2();
val schema: Schema = output.getSchema();
val reader: SpecificDatumReader[RecursiveRecord2] = new SpecificDatumReader[RecursiveRecord2](schema);
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
output = reader.read(output, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null": null}}"""));
{code}
{code:title=scala-repl-generic-2|borderStyle=solid}
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericDatumReader;

val parser = new Schema.Parser()
val schema: Schema = parser.parse("""{"type": "record", "name": "RecursiveRecord2", "fields": [{"name": "child", "type": ["RecursiveRecord2", "null"]}]}""");
val reader: GenericDatumReader[java.lang.Object] = new GenericDatumReader[java.lang.Object](schema);
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": null}"""));
val output = reader.read(null, DecoderFactory.get().jsonDecoder(schema, """{"child": {"null": null}}"""));
{code}

For both specific and generic, {{RecursiveRecord2}} works properly: it produces an object with recursive type and {{child == null}}.

My understanding of the official schema is that only {{RecursiveRecord2}} should be allowed to have a null {{child}}, so the JSON I supplied would not have been valid input for {{RecursiveRecord}}.  (If so, then it wouldn't even be possible to give it valid finite input.)  However, it should give a different error than a stack overflow, something to tell me that {{{"child": null}}} is not legal unless field {{child}} is declared as a union that includes {{null}}.

The reason one might want this (recursively defined types) is to make trees.  The example I gave had only one child for simplicity (i.e. it was a linked list), but the error would apply to binary trees as well.  For instance, here's a three-node list (a little cumbersome in JSON):

{code:title=motivating-example|borderStyle=solid}
{"child": {"RecursiveRecord2": {"child": {"RecursiveRecord2": {"child": null}}}}}
{code}

I haven't tested this in Avro deserialization (which would be a more reasonable use-case), but I don't know of a way to generate the Avro-encoded data without first getting it from human-typable JSON.  (I'm not constructing the Avro byte stream by hand.)


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)