drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-4824) JSON with complex nested data produces incorrect output with missing fields
Date Wed, 08 Feb 2017 00:49:42 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857131#comment-15857131
] 

Paul Rogers edited comment on DRILL-4824 at 2/8/17 12:49 AM:
-------------------------------------------------------------

[~sharnyk], with your change, the expected result would be the following (ignore spacing):

{code}
{ }
{ "Field1" : { "InnerField2": {"key2":"value2"} } }
{code}

What the user probably actually expects is:

{code}
{ "Field1" : { } }
{ "Field1" : { "InnerField2": {"key2":"value2"} } }
{code}

To understand why, we need to review why Drill violates the JSON standard. The following four
cases area all different in JSON:

{code}
{ "foo": null }
{ "foo": {} }
{ "foo": [] }
{ }
{code}

That is, a field can be:

* Null
* A map (possibly empty)
* An array (possibly empty)
* Not present

We do not support:

* Null map (not a JSON concept, in JSON anything null is still null.)
* Null array (as above)
* Not present

So we try to force fit these cases into what we do support. It used to be that we handle "not
present" to mean "present but null or empty." Your change says that an empty array or empty
map is the same as not present. And, of course, we still don't support null maps or arrays.
We treat them as empty, and so, now, as not present.

This is quite a difficult situation!

So, we seem to be adding rules that says:

* If a map or array is empty, omit it from the output.
* If a map or array contains only omitted members, then omit the outer map or array.

This means that all the following inputs produce the same empty output:

{code}
{ }
{ "field1": { } }
{ "field1": { "field1a": { } }
{ "field1": { "field1a": { "field1aA": { } } }
{ "field1": { "field1b": [ ] }
{ "field1": { "field1c": null }
{ "field2": [ ] }
{ "field2": [ { } ] }
{ "field2": [ { }, { } ] }
{ "field3": [ null ] }
{ "field4": null }
{code}

Your fix might help some users, but it is not proper JSON, so it will end up breaking behavior
for other users.

One way to resolve this is with a session option. This is not a proper solution, it just pushes
the problem onto the user.

{code}
ALTER SESSION SET something.json.hide-empty = true
{code}

Provides your behavior, while:

{code}
ALTER SESSION SET something.json.hide-empty = false
{code}

Keeps the old behavior.

What we really need is:

{code}
ALTER SESSION SET something.json.follow-the-standard = true
{code}

But, sadly, following the standard requires work far beyond the scope of this bug fix.


was (Author: paul-rogers):
[~sharnyk], with your change, the expected result would be the following (ignore spacing):

{code}
{ "Field1" : { } }
{ "Field1" : { "InnerField2": {"key2":"value2"} } }
{code}

What the user probably actually expects is:

{code}
{ }
{ "Field1" : { "InnerField2": {"key2":"value2"} } }
{code}

To understand why, we need to review why Drill violates the JSON standard. The following four
cases area all different in JSON:

{code}
{ "foo": null }
{ "foo": {} }
{ "foo": [] }
{ }
{code}

That is, a field can be:

* Null
* A map (possibly empty)
* An array (possibly empty)
* Not present

We do not support:

* Null map (not a JSON concept, in JSON anything null is still null.)
* Null array (as above)
* Not present

So we try to force fit these cases into what we do support. It used to be that we handle "not
present" to mean "present but null or empty." Your change says that an empty array or empty
map is the same as not present. And, of course, we still don't support null maps or arrays.
We treat them as empty, and so, now, as not present.

This is quite a difficult situation!

So, we seem to be adding rules that says:

* If a map or array is empty, omit it from the output.
* If a map or array contains only omitted members, then omit the outer map or array.

This means that all the following inputs produce the same empty output:
{ }
{ "field1": { } }
{ "field1": { "field1a": { } }
{ "field1": { "field1a": { "field1aA": { } } }
{ "field1": { "field1b: [ ] }
{ "field1": { "field1c": null }
{ "field2": [ ] }
{ "field2": [ { } ] }
{ "field2": [ { }, { } ] }
{ "field3": [ null ] }
{ "field4": null }
{code}

Your fix might help some users, but it is not proper JSON, so it will end up breaking behavior
for other users.

One way to resolve this is with a session option. This is not a proper solution, it just pushes
the problem onto the user.

{code}
ALTER SESSION SET something.json.hide-empty = true
{code}

Provides your behavior, while:

{code}
ALTER SESSION SET something.json.hide-empty = false
{code}

Keeps the old behavior.

What we really need is:

{code}
ALTER SESSION SET something.json.follow-the-standard = true
{code}

But, sadly, following the standard requires work far beyond the scope of this bug fix.

> JSON with complex nested data produces incorrect output with missing fields
> ---------------------------------------------------------------------------
>
>                 Key: DRILL-4824
>                 URL: https://issues.apache.org/jira/browse/DRILL-4824
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - JSON
>    Affects Versions: 1.0.0
>            Reporter: Roman
>            Assignee: Serhii Harnyk
>
> There is incorrect output in case of JSON file with complex nested data.
> _JSON:_
> {code:none|title=example.json|borderStyle=solid}
> {
>         "Field1" : {
>         }
> }
> {
>         "Field1" : {
>                 "InnerField1": {"key1":"value1"},
>                 "InnerField2": {"key2":"value2"}
>         }
> }
> {
>         "Field1" : {
>                 "InnerField3" : ["value3", "value4"],
>                 "InnerField4" : ["value5", "value6"]
>         }
> }
> {code}
> _Query:_
> {code:sql}
> select Field1 from dfs.`/tmp/example.json`
> {code}
> _Incorrect result:_
> {code:none}
> +---------------------------+
> |          Field1           |
> +---------------------------+
> {"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{"key1":"value1"},"InnerField2" {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--------------------------+
> {code}
> Theres is no need to output missing fields. In case of deeply nested structure we will
get unreadable result for user.
> _Correct result:_
> {code:none}
> +--------------------------+
> |         Field1           |
> +--------------------------+
> |{}                                                                     
> {"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
> {"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--------------------------+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message