Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5EB801880F for ; Wed, 2 Mar 2016 11:02:47 +0000 (UTC) Received: (qmail 56443 invoked by uid 500); 2 Mar 2016 11:02:43 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 56328 invoked by uid 500); 2 Mar 2016 11:02:42 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 56318 invoked by uid 99); 2 Mar 2016 11:02:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2016 11:02:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 71266C00ED for ; Wed, 2 Mar 2016 11:02:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id QvDkBiZsh825 for ; Wed, 2 Mar 2016 11:02:41 +0000 (UTC) Received: from mail-ob0-f169.google.com (mail-ob0-f169.google.com [209.85.214.169]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id BF4E95F341 for ; Wed, 2 Mar 2016 11:02:40 +0000 (UTC) Received: by mail-ob0-f169.google.com with SMTP id ts10so194763141obc.1 for ; Wed, 02 Mar 2016 03:02:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to; bh=M7kjxkdHMehghZQ80/RWDSw5wddJ4271P31fP8opYtU=; b=U+Z87U5CDYZeI+xrPYWKzk1x5oYDydyhUybEyLSltkkkA05Q6VajAcNfPSBvoWyzMZ d19Vq5LpKs8u6oAV6xoaRZs0tJ6vDseFmd/RHxV6ivByaAA1oQIbl4R16sz20w7V2qUd SUKsG//DdRF2jQql7JwcACwLiRF9cLx9UNt0rWZAii/h3dMY3VMljcou+DUj5C4QdZ2M bcUitxbLu5f/hN95k23QgqOHv7Zyfr28eYefUH4AfqF7Nl5Fb1Wqlb3UThS1ZetxU3TO oU7f8jv7FrS7PYWRrjIij3DyKA9nFi/G+whUGwW9L5rZAVg8tjW4neiwiwaj60KrlXmo qr3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=M7kjxkdHMehghZQ80/RWDSw5wddJ4271P31fP8opYtU=; b=YmxvQElsXTkFl/9zhzfa7xeGiR4n1yy3Ph97nJon0J1jFJD4g3CfUqFDqVLbL6/n6e 2ywQNgdr9RRWAvDalYygiwHFsXNQ5hA7leuApfKqg1XkQQ3+QAYU12raGFDDsTfyYLBg ZPX90vXJGb6VQvJ4DFi2mQ35rrvb7zEDUQgwPhRVP4+WTa5RPX/uMEi12yO8wcr9kcmh voOPt301blHRXGzPpnozrZI8iigJFtaJeOprTUky3i66uF4L/q/Lzv5/VPMvL1tlz52C qUCfRb8pdnCPVfEShmslT0h6NmW11rPrkz5yNFyntyNpNOQeNd923IgDEnHy3lqvsZmm wWsQ== X-Gm-Message-State: AD7BkJKu4NuFGAc9USbtzH+6/YwjEHEP5MrsBcgOyJQefAKMDU/4tJUXOVdYQM8qaxD0l9xe+SF6vEsZu8QKdA== MIME-Version: 1.0 X-Received: by 10.182.33.38 with SMTP id o6mr21192731obi.40.1456916554525; Wed, 02 Mar 2016 03:02:34 -0800 (PST) Received: by 10.157.40.51 with HTTP; Wed, 2 Mar 2016 03:02:34 -0800 (PST) Date: Wed, 2 Mar 2016 03:02:34 -0800 Message-ID: Subject: spark streaming From: Vinti Maheshwari To: user Content-Type: multipart/alternative; boundary=089e011838f0b33448052d0ed16d --089e011838f0b33448052d0ed16d Content-Type: text/plain; charset=UTF-8 Hi All, I wanted to set *StorageLevel.MEMORY_AND_DISK_SER* in my spark-streaming program as currently i am getting MetadataFetchFailedException*. *I am not sure where i should pass StorageLevel.MEMORY_AND_DISK, as it seems like createDirectStream doesn't allow to pass that parameter. val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder]( ssc, kafkaParams, topicsSet) Full Error: *org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 0* at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:460) at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:456) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:456) at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:183) at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:47) at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69) at org.apache.spark.rdd.RDD.iterator(RDD.scala:262) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ) Thanks, ~Vinti --089e011838f0b33448052d0ed16d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi All,

I wanted = to set StorageLevel.MEMORY_AND_DISK_SER in my spark-streaming = program as currently i am getting
MetadataFetchFaile= dException. I am not sure where i should pass StorageLevel.MEMORY_AND_DISK, as it seems like createDirect= Stream doesn't allow to pass that parameter.

val messages =3D KafkaUtils.createDirectStream[String<=
/span>, String, StringDecoder,=
 StringDecoder](
  ssc, kafkaParams, topicsSet)

Full Error:
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an outp= ut location for shuffle 0
=C2=A0=C2=A0=C2=A0 at org.apache.spark.Map= OutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuse= s$2.apply(MapOutputTracker.scala:460)
=C2=A0=C2=A0=C2=A0 at org.apache.s= park.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMa= pStatuses$2.apply(MapOutputTracker.scala:456)
=C2=A0=C2=A0=C2=A0 at scal= a.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(Traversabl= eLike.scala:772)
=C2=A0=C2=A0=C2=A0 at scala.collection.IndexedSeqOptimi= zed$class.foreach(IndexedSeqOptimized.scala:33)
=C2=A0=C2=A0=C2=A0 at sc= ala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
=C2=A0= =C2=A0=C2=A0 at scala.collection.TraversableLike$WithFilter.foreach(Travers= ableLike.scala:771)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.MapOutputTrac= ker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker= .scala:456)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.MapOutputTracker.getM= apSizesByExecutorId(MapOutputTracker.scala:183)
=C2=A0=C2=A0=C2=A0 at or= g.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:= 47)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.ShuffledRDD.compute(Shuff= ledRDD.scala:90)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.computeO= rReadCheckpoint(RDD.scala:300)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.rd= d.RDD.iterator(RDD.scala:264)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd= .MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
=C2=A0=C2=A0=C2=A0 = at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
=C2= =A0=C2=A0=C2=A0 at org.apache.spark.CacheManager.getOrCompute(CacheManager.= scala:69)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.iterator(RDD.sc= ala:262)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.scheduler.ResultTask.run= Task(ResultTask.scala:66)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.schedul= er.Task.run(Task.scala:88)
=C2=A0=C2=A0=C2=A0 at org.apache.spark.execut= or.Executor$TaskRunner.run(Executor.scala:214)
=C2=A0=C2=A0=C2=A0 at jav= a.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145= )
=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= un(ThreadPoolExecutor.java:615)
=C2=A0=C2=A0=C2=A0 at java.lang.Thread.r= un(Thread.java:745)

)

Thanks,
~Vinti
--089e011838f0b33448052d0ed16d--