spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject spark git commit: [SPARK-10955] [STREAMING] Add a warning if dynamic allocation for Streaming applications
Date Fri, 09 Oct 2015 01:53:57 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.5 ba601b1ac -> 3df750029


[SPARK-10955] [STREAMING] Add a warning if dynamic allocation for Streaming applications

Dynamic allocation can be painful for streaming apps and can lose data. Log a warning for
streaming applications if dynamic allocation is enabled.

Author: Hari Shreedharan <hshreedharan@apache.org>

Closes #8998 from harishreedharan/ss-log-error and squashes the following commits:

462b264 [Hari Shreedharan] Improve log message.
2733d94 [Hari Shreedharan] Minor change to warning message.
eaa48cc [Hari Shreedharan] Log a warning instead of failing the application if dynamic allocation
is enabled.
725f090 [Hari Shreedharan] Add config parameter to allow dynamic allocation if the user explicitly
sets it.
b3f9a95 [Hari Shreedharan] Disable dynamic allocation and kill app if it is enabled.
a4a5212 [Hari Shreedharan] [streaming] SPARK-10955. Disable dynamic allocation for Streaming
applications.

(cherry picked from commit 09841290055770a619a2e72fbaef1a5e694916ae)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3df75002
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3df75002
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3df75002

Branch: refs/heads/branch-1.5
Commit: 3df7500299030cf4e002591cb3af5804aa4563da
Parents: ba601b1
Author: Hari Shreedharan <hshreedharan@apache.org>
Authored: Thu Oct 8 18:53:38 2015 -0700
Committer: Tathagata Das <tathagata.das1565@gmail.com>
Committed: Thu Oct 8 18:53:52 2015 -0700

----------------------------------------------------------------------
 .../scala/org/apache/spark/streaming/StreamingContext.scala | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/3df75002/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
----------------------------------------------------------------------
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
index f4f0869..bcd98ea 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
@@ -44,7 +44,7 @@ import org.apache.spark.streaming.dstream._
 import org.apache.spark.streaming.receiver.{ActorReceiver, ActorSupervisorStrategy, Receiver}
 import org.apache.spark.streaming.scheduler.{JobScheduler, StreamingListener}
 import org.apache.spark.streaming.ui.{StreamingJobProgressListener, StreamingTab}
-import org.apache.spark.util.{CallSite, ShutdownHookManager, ThreadUtils}
+import org.apache.spark.util.{CallSite, ShutdownHookManager, ThreadUtils, Utils}
 
 /**
  * Main entry point for Spark Streaming functionality. It provides methods used to create
@@ -570,6 +570,13 @@ class StreamingContext private[streaming] (
           )
       }
     }
+
+    if (Utils.isDynamicAllocationEnabled(sc.conf)) {
+      logWarning("Dynamic Allocation is enabled for this application. " +
+        "Enabling Dynamic allocation for Spark Streaming applications can cause data loss
if " +
+        "Write Ahead Log is not enabled for non-replayable sources like Flume. " +
+        "See the programming guide for details on how to enable the Write Ahead Log")
+    }
   }
 
   /**


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message