From reviews-return-619471-archive-asf-public=cust-asf.ponee.io@spark.apache.org  Tue Feb 27 11:03:06 2018
Return-Path: <reviews-return-619471-archive-asf-public=cust-asf.ponee.io@spark.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by mx-eu-01.ponee.io (Postfix) with SMTP id C586F18066D
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 27 Feb 2018 11:03:05 +0100 (CET)
Received: (qmail 1078 invoked by uid 500); 27 Feb 2018 10:03:04 -0000
Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:reviews-help@spark.apache.org>
List-Unsubscribe: <mailto:reviews-unsubscribe@spark.apache.org>
List-Post: <mailto:reviews@spark.apache.org>
List-Id: <reviews.spark.apache.org>
Delivered-To: mailing list reviews@spark.apache.org
Received: (qmail 973 invoked by uid 99); 27 Feb 2018 10:03:04 -0000
Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Feb 2018 10:03:04 +0000
Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33)
	id 25B51EB4BA; Tue, 27 Feb 2018 10:03:04 +0000 (UTC)
From: jinxing64 <git@git.apache.org>
To: reviews@spark.apache.org
Reply-To: reviews@spark.apache.org
Message-ID: <git-pr-20685-spark@git.apache.org>
Subject: [GitHub] spark pull request #20685: [SPARK-23524] Big local shuffle blocks should not...
Content-Type: text/plain
Date: Tue, 27 Feb 2018 10:03:04 +0000 (UTC)

GitHub user jinxing64 opened a pull request:

    https://github.com/apache/spark/pull/20685

    [SPARK-23524] Big local shuffle blocks should not be checked for corruption.

    ## What changes were proposed in this pull request?
    
    In current code, all local blocks will be checked for corruption no matter it's big or not.  The reasons are as below:
    
    Size in FetchResult for local block is set to be 0 (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L327)
    SPARK-4105 meant to only check the small blocks(size<maxBytesInFlight/3), but for reason 1, below check will be invalid. https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L420
    
    We can fix this and avoid the OOM.
    
    ## How was this patch tested?
    
    UT added

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jinxing64/spark SPARK-23524

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20685
    
----
commit 535916c045b123e803c0f6dbf786076045036167
Author: jx158167 <jx158167@...>
Date:   2018-02-27T09:56:38Z

    [SPARK-23524] Big local shuffle blocks should not be checked for corruption.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org