spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ignacio Blasco <elnopin...@gmail.com>
Subject Re: RDD[Future[T]] => Future[RDD[T]]
Date Sun, 26 Jul 2015 20:25:40 GMT
Maybe using mapPartitions and .sequence inside it?
El 26/7/2015 10:22 p. m., "Ayoub" <benali.ayoub.info@gmail.com> escribió:

> Hello,
>
> I am trying to convert the result I get after doing some async IO :
>
> val rdd: RDD[T] = // some rdd
>
> val result: RDD[Future[T]] = rdd.map(httpCall)
>
> Is there a way collect all futures once they are completed in a *non
> blocking* (i.e. without scala.concurrent
> Await) and lazy way?
>
> If the RDD was a standard scala collection then calling
> "scala.concurrent.Future.sequence" would have resolved the issue but RDD is
> not a TraversableOnce (which is required by the method).
>
> Is there a way to do this kind of transformation with an RDD[Future[T]] ?
>
> Thanks,
> Ayoub.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Future-T-Future-RDD-T-tp24000.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message