spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Reading back hdfs files saved as case class
Date Fri, 07 Oct 2016 19:23:13 GMT
You can use the Dataset API -- it should solve this issue for case classes
that are not very complex.

On Fri, Oct 7, 2016 at 12:20 PM, Deepak Sharma <deepakmca05@gmail.com>
wrote:

> Hi
> I am saving RDD[Example] in hdfs from spark program , where Example is
> case class.
> Now when i am trying to read it back , it returns RDD[String] with the
> content as below:
> *Example(1,name,value)*
>
> The workaround can be to write as a string in hdfs and read it back as
> string and perform further processing.This way the case class name wouldn't
> appear at all in the file being written in hdfs.
> But i am keen to know if we can read the data directly in Spark if the
> RDD[Case_Class] is written to hdfs?
>
> --
> Thanks
> Deepak
>

Mime
View raw message