Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D2978200D70 for ; Fri, 22 Dec 2017 06:38:39 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D14A0160C2C; Fri, 22 Dec 2017 05:38:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 22289160C2B for ; Fri, 22 Dec 2017 06:38:38 +0100 (CET) Received: (qmail 15214 invoked by uid 500); 22 Dec 2017 05:38:38 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@spark.apache.org Received: (qmail 15205 invoked by uid 99); 22 Dec 2017 05:38:38 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Dec 2017 05:38:38 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2D5DEDFC2F; Fri, 22 Dec 2017 05:38:38 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: lixiao@apache.org To: commits@spark.apache.org Message-Id: <7e36d1d55a914b3c881f591e9e612109@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: spark git commit: [SPARK-22862] Docs on lazy elimination of columns missing from an encoder Date: Fri, 22 Dec 2017 05:38:38 +0000 (UTC) archived-at: Fri, 22 Dec 2017 05:38:40 -0000 Repository: spark Updated Branches: refs/heads/branch-2.2 1e4cca02f -> 1cf3e3a26 [SPARK-22862] Docs on lazy elimination of columns missing from an encoder This behavior has confused some users, so lets clarify it. Author: Michael Armbrust Closes #20048 from marmbrus/datasetAsDocs. (cherry picked from commit 8df1da396f64bb7fe76d73cd01498fdf3b8ed964) Signed-off-by: gatorsmile Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1cf3e3a2 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1cf3e3a2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1cf3e3a2 Branch: refs/heads/branch-2.2 Commit: 1cf3e3a26961d306eb17b7629d8742a4df45f339 Parents: 1e4cca0 Author: Michael Armbrust Authored: Thu Dec 21 21:38:16 2017 -0800 Committer: gatorsmile Committed: Thu Dec 21 21:38:34 2017 -0800 ---------------------------------------------------------------------- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/1cf3e3a2/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala index 1acbad9..fb700a4 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala @@ -368,6 +368,10 @@ class Dataset[T] private[sql]( * If the schema of the Dataset does not match the desired `U` type, you can use `select` * along with `alias` or `as` to rearrange or rename as required. * + * Note that `as[]` only changes the view of the data that is passed into typed operations, + * such as `map()`, and does not eagerly project away any columns that are not present in + * the specified class. + * * @group basic * @since 1.6.0 */ --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org For additional commands, e-mail: commits-help@spark.apache.org