spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cjwang>
Subject Finding previous and next element in a sorted RDD
Date Fri, 22 Aug 2014 00:42:41 GMT
I have an RDD containing elements sorted in certain order.  I would like to
map over the elements knowing the values of their respective previous and
next elements.

With regular List, I used to do this:  ("input" is a List below)

// The first of the previous measures and the last of the next measures are
val dummy = new Measure()
val ml = (dummy :: input) :+ dummy
// Take 3 element at the time.  Then 1st is the previous, the middle is the
current, and
// the last is the next.			
ml.iterator.sliding(3).map( tri => produceMeasure(tri.head, tri.tail.head,
tri.last) ) toList

Now, in RDD, how do I do that (elegantly I hope)?  I thought about zipping
an RDD with shifted self, but there are some messy partition issues about

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message