spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gurwls...@apache.org
Subject spark git commit: [SPARK-22484][DOC] Document PySpark DataFrame csv writer behavior whe…
Date Tue, 28 Nov 2017 01:14:39 GMT
Repository: spark
Updated Branches:
  refs/heads/master 087879a77 -> 33d43bf1b


[SPARK-22484][DOC] Document PySpark DataFrame csv writer behavior whe…

## What changes were proposed in this pull request?

In PySpark API Document, DataFrame.write.csv() says that setting the quote parameter to an
empty string should turn off quoting. Instead, it uses the [null character](https://en.wikipedia.org/wiki/Null_character)
as the quote.

This PR fixes the doc.

## How was this patch tested?

Manual.

```
cd python/docs
make html
open _build/html/pyspark.sql.html
```

Author: gaborgsomogyi <gabor.g.somogyi@gmail.com>

Closes #19814 from gaborgsomogyi/SPARK-22484.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/33d43bf1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/33d43bf1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/33d43bf1

Branch: refs/heads/master
Commit: 33d43bf1b6f55594187066f0e38ba3985fa2542b
Parents: 087879a
Author: gaborgsomogyi <gabor.g.somogyi@gmail.com>
Authored: Tue Nov 28 10:14:35 2017 +0900
Committer: hyukjinkwon <gurwls223@gmail.com>
Committed: Tue Nov 28 10:14:35 2017 +0900

----------------------------------------------------------------------
 python/pyspark/sql/readwriter.py                                  | 3 +--
 .../src/main/scala/org/apache/spark/sql/DataFrameWriter.scala     | 3 ++-
 2 files changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/33d43bf1/python/pyspark/sql/readwriter.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index a75bdf8..1ad974e 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -828,8 +828,7 @@ class DataFrameWriter(OptionUtils):
                     set, it uses the default value, ``,``.
         :param quote: sets the single character used for escaping quoted values where the
                       separator can be part of the value. If None is set, it uses the default
-                      value, ``"``. If you would like to turn off quotations, you need to
set an
-                      empty string.
+                      value, ``"``. If an empty string is set, it uses ``u0000`` (null character).
         :param escape: sets the single character used for escaping quotes inside an already
                        quoted value. If None is set, it uses the default value, ``\``
         :param escapeQuotes: a flag indicating whether values containing quotes should always

http://git-wip-us.apache.org/repos/asf/spark/blob/33d43bf1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index e3fa2ce..35abecc 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -592,7 +592,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
    * <li>`sep` (default `,`): sets the single character as a separator for each
    * field and value.</li>
    * <li>`quote` (default `"`): sets the single character used for escaping quoted
values where
-   * the separator can be part of the value.</li>
+   * the separator can be part of the value. If an empty string is set, it uses `u0000`
+   * (null character).</li>
    * <li>`escape` (default `\`): sets the single character used for escaping quotes
inside
    * an already quoted value.</li>
    * <li>`escapeQuotes` (default `true`): a flag indicating whether values containing


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message