spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cjwang ...@cjwang.us>
Subject Finding previous and next element in a sorted RDD
Date Fri, 22 Aug 2014 00:42:41 GMT
I have an RDD containing elements sorted in certain order.  I would like to
map over the elements knowing the values of their respective previous and
next elements.

With regular List, I used to do this:  ("input" is a List below)

// The first of the previous measures and the last of the next measures are
dummy
val dummy = new Measure()
val ml = (dummy :: input) :+ dummy
		
// Take 3 element at the time.  Then 1st is the previous, the middle is the
current, and
// the last is the next.			
ml.iterator.sliding(3).map( tri => produceMeasure(tri.head, tri.tail.head,
tri.last) ) toList

Now, in RDD, how do I do that (elegantly I hope)?  I thought about zipping
an RDD with shifted self, but there are some messy partition issues about
zip.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Finding-previous-and-next-element-in-a-sorted-RDD-tp12621.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message