spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Hakobian <nicholas.hakob...@rallyhealth.com>
Subject Re: Custom delimiter file load
Date Sat, 31 Dec 2016 21:48:38 GMT
See the documentation for the options given to the csv function:
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameReader@csv(paths:String*):org.apache.spark.sql.DataFrame

The options can be passed with the option/options functions to the
DataFrameReader class (a similar syntax is also available in pySpark).

-Nick


Nicholas Szandor Hakobian, Ph.D.
Senior Data Scientist
Rally Health
nicholas.hakobian@rallyhealth.com


On Sat, Dec 31, 2016 at 9:58 AM, A Shaikh <shaikh.afzal@gmail.com> wrote:

> In Pyspark 2 loading file wtih any delimiter into Dataframe is pretty
> straightforward
> spark.read.csv(file, schema=, sep='|')
>
> Is there something similar in Spark 2 in Scala! spark.read.csv(path,
> sep='|')?
>
>

Mime
View raw message