spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <cloud0...@gmail.com>
Subject Re: A question about rdd transformation
Date Fri, 23 Jun 2017 06:12:18 GMT
<html><head></head><body dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space;"><meta http-equiv="Content-Type" content="text/html
charset=us-ascii"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class=""><div class="">The exception message should include the&nbsp;<span
class="">lineage</span><span class="">&nbsp;of the un-serializable object,
can you post that too?</span></div><span class=""><br class=""></span><div><blockquote
type="cite" class=""><div class="">On 23 Jun 2017, at 11:23 AM, Lionel Luffy &lt;<a
href="mailto:lioneluffy@gmail.com" class="">lioneluffy@gmail.com</a>&gt; wrote:</div><br
class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">add
dev list. Who can help on below question?<div class=""><br class=""></div><div
class="">Thanks &amp; Best Regards,</div><div class="">LL<br class=""><div
class=""><br class=""><div class="gmail_quote">---------- Forwarded message
----------<br class="">From: <b class="gmail_sendername">Lionel Luffy</b>
<span dir="ltr" class="">&lt;<a href="mailto:lioneluffy@gmail.com" class="">lioneluffy@gmail.com</a>&gt;</span><br
class="">Date: Fri, Jun 23, 2017 at 11:20 AM<br class="">Subject: Re: A question
about rdd transformation<br class="">To: <a href="mailto:user@spark.apache.org" class="">user@spark.apache.org</a><br
class=""><br class=""><br class=""><div dir="ltr" class="">Now I found
the root cause is a Wrapper class in AnyRef is not Serializable, but even though I changed
it to implements Serializable. the 'rows' still cannot get data... Any suggestion?</div><div
class="HOEnZb"><div class="h5"><div class="gmail_extra"><br class=""><div
class="gmail_quote">On Fri, Jun 23, 2017 at 10:56 AM, Lionel Luffy <span dir="ltr" class="">&lt;<a
href="mailto:lioneluffy@gmail.com" target="_blank" class="">lioneluffy@gmail.com</a>&gt;</span>
wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"><div dir="ltr" class="">Hi there,<div class="">I'm
trying to do below action while it always return <a href="http://java.io/" target="_blank"
class="">java.io</a>.NotSerializableExcepti<wbr class="">on in the shuffle
task.</div><div class="">I've checked that Array is serializable. how can I get
the data of rdd in newRDD?</div><div class=""><br class=""></div><div
class="">step 1: val rdd: RDD[(AnyRef, Array[AnyRef]] {......}</div><div class=""><br
class=""></div><div class="">step2 : &nbsp; &nbsp; &nbsp; rdd</div><div
class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;.partitionBy(partitioner)</div><div class="">&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;.map(_._2)</div><div
class=""><br class=""></div><div class="">step3: &nbsp;pass rdd to
newRDD as prev:</div><div class="">newRDD[K, V] (</div><div class="">xxx,</div><div
class="">xxx,</div><div class="">xxx,</div><div class="">prev:
RDD[Array[AnyRef]] extends RDD[(K, V)] (prev) {</div><div class=""><br class=""></div><div
class="">override protected def getPartitions() {...}</div><div class=""><br
class=""></div><div class="">override def compute(split: Partition, context:
TaskContext): Iterator[(K, V)] {...</div><div class="">&nbsp; &nbsp; &nbsp;
val rows = firstParent[Array[AnyRef]].ite<wbr class="">rator(split, context)</div><div
class=""><br class=""></div><div class="">&nbsp; &nbsp;}</div><div
class=""><br class=""></div><div class="">}</div><div class=""><br
class=""></div><div class=""><br class=""></div><div class="">Thanks,</div><div
class="">LL</div></div>
</blockquote></div><br class=""></div>
</div></div></div><br class=""></div></div></div>
</div></blockquote></div><br class=""></div></body></html>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message