From reviews-return-713854-archive-asf-public=cust-asf.ponee.io@spark.apache.org Tue Oct 9 22:33:57 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 21D6418077B for ; Tue, 9 Oct 2018 22:33:56 +0200 (CEST) Received: (qmail 34884 invoked by uid 500); 9 Oct 2018 20:33:56 -0000 Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@spark.apache.org Received: (qmail 34542 invoked by uid 99); 9 Oct 2018 20:33:55 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Oct 2018 20:33:55 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 5BBC2DFF2E; Tue, 9 Oct 2018 20:33:55 +0000 (UTC) From: vanzin To: reviews@spark.apache.org Reply-To: reviews@spark.apache.org References: In-Reply-To: Subject: [GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn... Content-Type: text/plain Message-Id: <20181009203355.5BBC2DFF2E@git1-us-west.apache.org> Date: Tue, 9 Oct 2018 20:33:55 +0000 (UTC) Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r223842417 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -806,6 +806,22 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock) } // Clean the blacklist from the expired entries. clearBlacklist(CLEAN_INTERVAL_S) + + // Delete driver logs from the configured spark dfs dir that exceed the configured max age + try { + val hdfsDir = conf.get("spark.driver.log.dfsDir") + val appDirs = fs.listLocatedStatus(new Path(hdfsDir)) + while (appDirs.hasNext()) { + val appDirStatus = appDirs.next() + if (appDirStatus.getModificationTime() < maxTime) { + logInfo(s"Deleting expired driver log for: ${appDirStatus.getPath().getName()}") + deleteLog(appDirStatus.getPath()) + } + } + } catch { + case nse: NoSuchElementException => // no-op --- End diff -- What call throws this exception? --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org For additional commands, e-mail: reviews-help@spark.apache.org