spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MaxGekk <>
Subject [GitHub] spark pull request #21999: [WIP][SQL] Flattening nested structures
Date Sat, 04 Aug 2018 20:55:18 GMT
GitHub user MaxGekk opened a pull request:

    [WIP][SQL] Flattening nested structures

    ## What changes were proposed in this pull request?
    In the PR, I propose new unary expression `StructFlatten` for flattening nested structures.
For example, a dataset with the schema:
     |-- st: struct (nullable = false)
     |    |-- col1: long (nullable = false)
     |    |-- col2: struct (nullable = false)
     |    |    |-- col3: long (nullable = false)
    by applying `struct_flatten(st)` it will be transformed to:
     |-- structflatten(st): struct (nullable = false)
     |    |-- col1: long (nullable = false)
     |    |-- col2_col3: long (nullable = false)
    ## How was this patch tested?
    Added new tests to `CollectionExpressionsSuite` to check flattening of 2-3 nested structures
and negative tests to be sure that `struct_flatten` doesn't affect other types.

You can merge this pull request into a Git repository by running:

    $ git pull struct_flatten

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21999
commit 5603918ae963f78aafb2d1f4f2bd9d566870495b
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T13:38:08Z

    Initial implementation

commit 0be0d059b8bf571068226c515888a64093468cff
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T16:07:45Z

    Making the depth and delimiter as parameters

commit 5666ec372a4b79f6161120584abc0c312b111bfb
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T18:04:23Z

    Test for depth = 0

commit cd88a2125ba6932ba1fdceca1a24d57124a23afa
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T18:21:19Z

    Test for depth = 1

commit b0da02d37ac6db38f63bac95dc295ac37fe4a692
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T18:30:18Z

    Renaming st to struct

commit ec361791b83d71f29823157a2c2b49162ddb5901
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T19:24:37Z

    Negative tests

commit ced63d7f093c168e2bc9457b6c08b87bfe6c0751
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T20:10:00Z

    Register struct_flatten

commit 5b568c67951f6f620cd0d549fdbd0c25f819fe43
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-08-04T20:42:00Z

    Merge remote-tracking branch 'origin/master' into struct_flatten
    # Conflicts:
    #	sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message