spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Does RDD[Type1, Iterable[Type2]] split into multiple partitions?
Date Thu, 10 Dec 2015 13:29:18 GMT
No, since the signature itself limits it.


On Thu, Dec 10, 2015 at 9:19 PM, JaeSung Jun <jaesjun@gmail.com> wrote:

> Hi,
>
> I'm currently working on Iterable type of RDD, which is like :
>
> val keyValueIterableRDD[CaseClass1, Iterable[CaseClass2]] = buildRDD(...)
>
> If there is only one unique key and Iterable is big enough, would this
> Iterable be partitioned across all executors like followings ?
>
> (executor1)
> (xxx, iterator from 0 to 10,000)
>
> (executor2)
> (xxx, iterator from 10,001 to 20,000)
>
> (executor2)
> (xxx, iterator from 20,001 to 30,000)
>
> ...
>
> Thanks
> Jason
>
>

Mime
View raw message