spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gengliangwang <>
Subject [GitHub] spark pull request #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIn...
Date Sun, 08 Apr 2018 18:56:23 GMT
GitHub user gengliangwang opened a pull request:

    [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

    ## What changes were proposed in this pull request?
    Currently `PartitioningAwareFileIndex` accepts an optional parameter `userPartitionSchema`.
If provided, it will combine the inferred partition schema with the parameter.
    1. to get `userPartitionSchema`, we need to  combine inferred partition schema with `userSpecifiedSchema`
    2. to get the inferred partition schema, we have to create a temporary file index.
    Only after that, a final version of `PartitioningAwareFileIndex` can be created.
    This can be improved by passing `userSpecifiedSchema` to `PartitioningAwareFileIndex`.
    With the improvement, we can reduce redundant code and avoid parsing the file partition
    ## How was this patch tested?
    Unit test

You can merge this pull request into a Git repository by running:

    $ git pull PartitioningAwareFileIndex

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21004
commit 35aff24743ff13ccd370a8e3747a3044e8a671c9
Author: Gengliang Wang <>
Date:   2018-04-08T18:19:48Z

    improve PartitioningAwareFileIndex



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message