spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vincent-grosbois <...@git.apache.org>
Subject [GitHub] spark pull request #21933: [SPARK-24917] make chunk size configurable
Date Tue, 31 Jul 2018 16:22:36 GMT
GitHub user vincent-grosbois opened a pull request:

    https://github.com/apache/spark/pull/21933

    [SPARK-24917] make chunk size configurable

    Add an option in Spark configuration to change the chunk size (which is by default 4 Mb).
    
    This would allow to bypass the issue mentionned in SPARK-24917 by allowing users to define
larger chunks
    
    ## What changes were proposed in this pull request?
    I'm introducing a configuration parameter to define the chunk size
    
    ## How was this patch tested?
    not tested as it's a trivial test. I can write a test if needed

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vincent-grosbois/spark chunk_size

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21933
    
----
commit 0251bd517e7fd3e695cb8366ffa03de8c9e2900b
Author: vincent-grosbois <vincent.grosbois@...>
Date:   2018-07-31T15:21:24Z

    [SPARK-24917] make chunk size configurable
    
    Add an option in Spark configuration to change the chunk size (which is by default 4 Mb).
    
    This would allow to bypass the issue mentionned in SPARK-24917 when fetching large partitions
(a bit less than 2 Gb)

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message