spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lix...@apache.org
Subject spark git commit: [SPARK-22862] Docs on lazy elimination of columns missing from an encoder
Date Fri, 22 Dec 2017 05:38:38 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 1e4cca02f -> 1cf3e3a26


[SPARK-22862] Docs on lazy elimination of columns missing from an encoder

This behavior has confused some users, so lets clarify it.

Author: Michael Armbrust <michael@databricks.com>

Closes #20048 from marmbrus/datasetAsDocs.

(cherry picked from commit 8df1da396f64bb7fe76d73cd01498fdf3b8ed964)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1cf3e3a2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1cf3e3a2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1cf3e3a2

Branch: refs/heads/branch-2.2
Commit: 1cf3e3a26961d306eb17b7629d8742a4df45f339
Parents: 1e4cca0
Author: Michael Armbrust <michael@databricks.com>
Authored: Thu Dec 21 21:38:16 2017 -0800
Committer: gatorsmile <gatorsmile@gmail.com>
Committed: Thu Dec 21 21:38:34 2017 -0800

----------------------------------------------------------------------
 sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/1cf3e3a2/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
index 1acbad9..fb700a4 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -368,6 +368,10 @@ class Dataset[T] private[sql](
    * If the schema of the Dataset does not match the desired `U` type, you can use `select`
    * along with `alias` or `as` to rearrange or rename as required.
    *
+   * Note that `as[]` only changes the view of the data that is passed into typed operations,
+   * such as `map()`, and does not eagerly project away any columns that are not present
in
+   * the specified class.
+   *
    * @group basic
    * @since 1.6.0
    */


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message