Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm
Precedence: bulk
From: kayousterhout <git@git.apache.org>
To: reviews@spark.apache.org
Reply-To: reviews@spark.apache.org
References: <git-pr-4749-spark@git.apache.org>
In-Reply-To: <git-pr-4749-spark@git.apache.org>
Subject: [GitHub] spark pull request: [SPARK-5982] Remove incorrect Local Read
 Time ...
Content-Type: text/plain
Message-Id: <20150224234938.547B0E0388@git1-us-west.apache.org>
Date: Tue, 24 Feb 2015 23:49:38 +0000 (UTC)

Github user kayousterhout commented on the pull request:

    https://github.com/apache/spark/pull/4749#issuecomment-75876725
  
    @ksakellis I did this in my own branch here: https://github.com/kayousterhout/spark-1/commit/eb84a0a2201d392a275179293f8a1c4c23981b24 and can submit a PR for Spark if there is sufficient interest.  We can't do it in unpackBlock; the problem is that the data gets read record-by-record, so we'd have to add how long it took to read each record (in unpackBlock, we still have have an iterator that points to a memory-mapped region, but the data still isn't necessarily read from disk there).  Any kind of per-record logging ends up being too high overhead.  The way I did it in the change I referred to is to add buffering to the reading of input data, and then you can log the time to read each chunk of the buffer (this is the same thing we do to track the time to write shuffle files to disk).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org