Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAOxYJ7nQJ8Ca8_FM26fL=SS=oFeXNvMt4e1j8BbJM9gZ6B5vcQ@mail.gmail.com>
References: <CAOxYJ7nQJ8Ca8_FM26fL=SS=oFeXNvMt4e1j8BbJM9gZ6B5vcQ@mail.gmail.com>
From: Nicholas Hakobian <nicholas.hakobian@rallyhealth.com>
Date: Sat, 31 Dec 2016 13:48:38 -0800
Message-ID: <CAMzev0cmTU0WfSTJYV71sViX5-NPZC8sM4sOXtZKzVXjaRe3Kg@mail.gmail.com>
Subject: Re: Custom delimiter file load
To: A Shaikh <shaikh.afzal@gmail.com>
Cc: "user @spark" <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a114022ca074e500544fb4807
archived-at: Sat, 31 Dec 2016 21:49:03 -0000

--001a114022ca074e500544fb4807
Content-Type: text/plain; charset=UTF-8

See the documentation for the options given to the csv function:
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameReader@csv(paths:String*):org.apache.spark.sql.DataFrame

The options can be passed with the option/options functions to the
DataFrameReader class (a similar syntax is also available in pySpark).

-Nick


Nicholas Szandor Hakobian, Ph.D.
Senior Data Scientist
Rally Health
nicholas.hakobian@rallyhealth.com


On Sat, Dec 31, 2016 at 9:58 AM, A Shaikh <shaikh.afzal@gmail.com> wrote:

> In Pyspark 2 loading file wtih any delimiter into Dataframe is pretty
> straightforward
> spark.read.csv(file, schema=, sep='|')
>
> Is there something similar in Spark 2 in Scala! spark.read.csv(path,
> sep='|')?
>
>

--001a114022ca074e500544fb4807
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">See the documentation for the options given to the csv fun=
ction:=C2=A0<a href=3D"http://spark.apache.org/docs/latest/api/scala/index.=
html#org.apache.spark.sql.DataFrameReader@csv(paths:String*):org.apache.spa=
rk.sql.DataFrame">http://spark.apache.org/docs/latest/api/scala/index.html#=
org.apache.spark.sql.DataFrameReader@csv(paths:String*):org.apache.spark.sq=
l.DataFrame</a><div><br></div><div>The options can be passed with the optio=
n/options functions to the DataFrameReader class (a similar syntax is also =
available in pySpark).</div><div><br></div><div>-Nick</div><div><br></div><=
div class=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_signat=
ure" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"l=
tr"><div><div dir=3D"ltr"><div><div dir=3D"ltr">Nicholas Szandor Hakobian, =
Ph.D.<div>Senior Data Scientist</div><div>Rally Health</div><div><a href=3D=
"mailto:nicholas.hakobian@rallyhealth.com" target=3D"_blank">nicholas.hakob=
ian@rallyhealth.com</a></div><div><br></div></div></div></div></div></div><=
/div></div></div></div>
<br><div class=3D"gmail_quote">On Sat, Dec 31, 2016 at 9:58 AM, A Shaikh <s=
pan dir=3D"ltr">&lt;<a href=3D"mailto:shaikh.afzal@gmail.com" target=3D"_bl=
ank">shaikh.afzal@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"g=
mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex"><div dir=3D"ltr">In Pyspark 2 loading file wtih any delimiter into=
 Dataframe is pretty straightforward<div>spark.read.csv(file, schema=3D, se=
p=3D&#39;|&#39;)<br></div><div><br></div><div>Is there something similar in=
 Spark 2 in Scala! spark.read.csv(path, sep=3D&#39;|&#39;)?</div><div><br><=
/div></div>
</blockquote></div><br></div></div>

--001a114022ca074e500544fb4807--