Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 17 Jul 2017 14:43:00 +0000 (UTC)
From: "eugen yushin (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13087599.1500302221000.263243.1500302580114@Atlassian.JIRA>
In-Reply-To: <JIRA.13087599.1500302221000@Atlassian.JIRA>
References: <JIRA.13087599.1500302221000@Atlassian.JIRA> <JIRA.13087599.1500302221066@jira-lw-us.apache.org>
Subject: [jira] [Updated] (SPARK-21442) Spark CSV writer trims trailing
 spaces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 17 Jul 2017 14:43:05 -0000


     [ https://issues.apache.org/jira/browse/SPARK-21442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

eugen yushin updated SPARK-21442:
---------------------------------
    Affects Version/s: 2.1.1

> Spark CSV writer trims trailing spaces
> --------------------------------------
>
>                 Key: SPARK-21442
>                 URL: https://issues.apache.org/jira/browse/SPARK-21442
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>    Affects Versions: 2.1.0, 2.1.1
>         Environment: Version 2.1.0-mapr-1703
> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_131)
>            Reporter: eugen yushin
>
> Looks like Spark truncates trailing spaces saving data with csv codec. Check the following example for more details (note extra space at the end of "Johny " field):
> {code}
> scala> case class SampleRow(field1: String, field2: String)
> defined class SampleRow
> scala> val fooDS = Seq(SampleRow("Johny ", "Doe"), SampleRow("Ivan", "Susanin")).toDS()
> fooDS: org.apache.spark.sql.Dataset[SampleRow] = [field1: string, field2: string]
> scala> fooDS.collect.foreach(println)
> SampleRow(Johny ,Doe)
> SampleRow(Ivan,Susanin)
> scala> fooDS.show()
> +------+-------+
> |field1| field2|
> +------+-------+
> |Johny |    Doe|
> |  Ivan|Susanin|
> +------+-------+
> scala> import org.apache.spark.sql.SaveMode
> import org.apache.spark.sql.SaveMode
> scala> fooDS.write.option("delimiter", "|").mode(SaveMode.Overwrite).csv("file:///tmp/spaces.txt")
> cat /tmp/spaces.txt/*
> Johny|Doe
> Ivan|Susanin
> {code}
> I expect space before the pipe at the first line in output file.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org