Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9F7F0200D11 for ; Mon, 18 Sep 2017 06:20:21 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9E0621609D9; Mon, 18 Sep 2017 04:20:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E33911609D8 for ; Mon, 18 Sep 2017 06:20:20 +0200 (CEST) Received: (qmail 20357 invoked by uid 500); 18 Sep 2017 04:20:20 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@spark.apache.org Received: (qmail 20347 invoked by uid 99); 18 Sep 2017 04:20:20 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Sep 2017 04:20:20 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E36FEE95BD; Mon, 18 Sep 2017 04:20:19 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: gurwls223@apache.org To: commits@spark.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles Date: Mon, 18 Sep 2017 04:20:19 +0000 (UTC) archived-at: Mon, 18 Sep 2017 04:20:21 -0000 Repository: spark Updated Branches: refs/heads/master 6308c65f0 -> 7c7266208 [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1000, in show_profiles self.profiler_collector.show_profiles() AttributeError: 'NoneType' object has no attribute 'show_profiles' >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles self.profiler_collector.dump_profiles(path) AttributeError: 'NoneType' object has no attribute 'dump_profiles' ``` to ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1003, in show_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. ``` ## How was this patch tested? Unit tests added in `python/pyspark/tests.py` and manual tests. Author: hyukjinkwon Closes #19260 from HyukjinKwon/profile-errors. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7c726620 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7c726620 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7c726620 Branch: refs/heads/master Commit: 7c7266208a3be984ac1ce53747dc0c3640f4ecac Parents: 6308c65 Author: hyukjinkwon Authored: Mon Sep 18 13:20:11 2017 +0900 Committer: hyukjinkwon Committed: Mon Sep 18 13:20:11 2017 +0900 ---------------------------------------------------------------------- python/pyspark/context.py | 12 ++++++++++-- python/pyspark/tests.py | 16 ++++++++++++++++ 2 files changed, 26 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/7c726620/python/pyspark/context.py ---------------------------------------------------------------------- diff --git a/python/pyspark/context.py b/python/pyspark/context.py index a704604..a33f6dc 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -997,12 +997,20 @@ class SparkContext(object): def show_profiles(self): """ Print the profile stats to stdout """ - self.profiler_collector.show_profiles() + if self.profiler_collector is not None: + self.profiler_collector.show_profiles() + else: + raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def dump_profiles(self, path): """ Dump the profile stats into directory `path` """ - self.profiler_collector.dump_profiles(path) + if self.profiler_collector is not None: + self.profiler_collector.dump_profiles(path) + else: + raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def getConf(self): conf = SparkConf() http://git-wip-us.apache.org/repos/asf/spark/blob/7c726620/python/pyspark/tests.py ---------------------------------------------------------------------- diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py index 3c108ec..da99872 100644 --- a/python/pyspark/tests.py +++ b/python/pyspark/tests.py @@ -1296,6 +1296,22 @@ class ProfilerTests(PySparkTestCase): rdd.foreach(heavy_foo) +class ProfilerTests2(unittest.TestCase): + def test_profiler_disabled(self): + sc = SparkContext(conf=SparkConf().set("spark.python.profile", "false")) + try: + self.assertRaisesRegexp( + RuntimeError, + "'spark.python.profile' configuration must be set", + lambda: sc.show_profiles()) + self.assertRaisesRegexp( + RuntimeError, + "'spark.python.profile' configuration must be set", + lambda: sc.dump_profiles("/tmp/abc")) + finally: + sc.stop() + + class InputFormatTests(ReusedPySparkTestCase): @classmethod --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org For additional commands, e-mail: commits-help@spark.apache.org