spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Sharma <deepakmc...@gmail.com>
Subject Re: What is the equivalent of forearchRDD in DataFrames?
Date Thu, 26 Oct 2017 12:40:39 GMT
df.rdd.foreach

Thanks
Deepak

On Oct 26, 2017 18:07, "Noorul Islam Kamal Malmiyoda" <noorul@noorul.com>
wrote:

> Hi all,
>
> I have a Dataframe with 1000 records. I want to split them into 100
> each and post to rest API.
>
> If it was RDD, I could use something like this
>
>     myRDD.foreachRDD {
>       rdd =>
>         rdd.foreachPartition {
>           partition => {
>
> This will ensure that code is executed on executors and not on driver.
>
> Is there any similar approach that we can take for Dataframes? I see
> examples on stackoverflow with collect() which will bring whole data
> to driver.
>
> Thanks and Regards
> Noorul
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message