spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang, Liye (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-4989) wrong application configuration cause cluster down in standalone mode
Date Tue, 30 Dec 2014 02:03:13 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260667#comment-14260667
] 

Zhang, Liye commented on SPARK-4989:
------------------------------------

Assume that we set *spark.eventLog.dir=hdfs://host:port/user/username/eventLog0, hdfs://host:port/user/username/eventLog1*,
when starting the application, application will fail due to the exception, like following:
{code}
Exception in thread "main" java.net.URISyntaxException: Illegal character in path at index
41: hdfs//host:port/user/username/eventLog0, hdfs://host:port/user/username/eventLog1
        at java.net.URI$Parser.fail(URI.java:2829)
        at java.net.URI$Parser.checkChars(URI.java:3002)
        at java.net.URI$Parser.parseHierarchical(URI.java:3086)
        at java.net.URI$Parser.parse(URI.java:3034)
        at java.net.URI.<init>(URI.java:595)
        at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:64)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:357)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:136)
        at com.intel.benchmark.wordcount.Driver$.main(Driver.scala:44)
        at com.intel.benchmark.wordcount.Driver.main(Driver.scala)
{code}

This will cause the *Master* to remove the application and rebuild the sparkUI. In which will
also throw the exception to *Master Actor*, which will lead to *Master*'s restart. The exception
will be like the following:
{code}
14/12/30 09:48:18 ERROR OneForOneStrategy: Illegal character in path at index 41: hdfs://sr148:9000/user/liyezhan/eventLog,
hdfs://sr148:9000/user/liyezhan/eventLog1
java.net.URISyntaxException: Illegal character in path at index 41: hdfs//host:port/user/username/eventLog0,
hdfs://host:port/user/username/eventLog1
  at java.net.URI$Parser.fail(URI.java:2829)
  at java.net.URI$Parser.checkChars(URI.java:3002)
  at java.net.URI$Parser.parseHierarchical(URI.java:3086)
  at java.net.URI$Parser.parse(URI.java:3034)
  at java.net.URI.<init>(URI.java:595)
  at org.apache.spark.util.Utils$.resolveURI(Utils.scala:1561)
  at org.apache.spark.scheduler.EventLoggingListener$.getLogPath(EventLoggingListener.scala:262)
  at org.apache.spark.deploy.master.Master$$anonfun$11.apply(Master.scala:724)
  at org.apache.spark.deploy.master.Master$$anonfun$11.apply(Master.scala:724)
  at scala.Option.map(Option.scala:145)
  at org.apache.spark.deploy.master.Master.rebuildSparkUI(Master.scala:724)
  at org.apache.spark.deploy.master.Master.removeApplication(Master.scala:695)
  at org.apache.spark.deploy.master.Master.finishApplication(Master.scala:673)
  at org.apache.spark.deploy.master.Master$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$29.apply(Master.scala:419)
  at org.apache.spark.deploy.master.Master$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$29.apply(Master.scala:419)
  at scala.Option.foreach(Option.scala:236)
  at org.apache.spark.deploy.master.Master$$anonfun$receiveWithLogging$1.applyOrElse(Master.scala:419)
  at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
  at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
  at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
  at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:53)
  at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:42)
  at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
  at org.apache.spark.util.ActorLogReceive$$anon$1.applyOrElse(ActorLogReceive.scala:42)
  at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
  at org.apache.spark.deploy.master.Master.aroundReceive(Master.scala:51)
  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
  at akka.actor.ActorCell.invoke(ActorCell.scala:487)
  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
  at akka.dispatch.Mailbox.run(Mailbox.scala:220)
  at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}

> wrong application configuration cause cluster down in standalone mode
> ---------------------------------------------------------------------
>
>                 Key: SPARK-4989
>                 URL: https://issues.apache.org/jira/browse/SPARK-4989
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy, Spark Core
>    Affects Versions: 1.0.0, 1.1.0, 1.2.0
>            Reporter: Zhang, Liye
>            Priority: Critical
>
> when enabling eventlog in standalone mode, if give the wrong configuration, the standalone
cluster will down (cause master restart, lose connection with workers). 
> How to reproduce: just give an invalid value to "spark.eventLog.dir", for example: *spark.eventLog.dir=hdfs://tmp/logdir1,
hdfs://tmp/logdir2*. This will throw illegalArgumentException, which will cause the *Master*
restart. And the whole cluster is not available.
> This is not acceptable that cluster is crashed by one application's wrong setting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message