Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EF0451783F for ; Mon, 2 Feb 2015 20:24:21 +0000 (UTC) Received: (qmail 46307 invoked by uid 500); 2 Feb 2015 20:24:15 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 46225 invoked by uid 500); 2 Feb 2015 20:24:15 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 46002 invoked by uid 99); 2 Feb 2015 20:24:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2015 20:24:11 +0000 X-ASF-Spam-Status: No, hits=2.5 required=10.0 tests=FREEMAIL_ENVFROM_END_DIGIT,SPF_SOFTFAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of ankitsoni9@gmail.com does not designate 162.253.133.43 as permitted sender) Received: from [162.253.133.43] (HELO mwork.nabble.com) (162.253.133.43) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2015 20:24:06 +0000 Received: from mben.nabble.com (unknown [162.253.133.72]) by mwork.nabble.com (Postfix) with ESMTP id 818961299845 for ; Mon, 2 Feb 2015 12:23:46 -0800 (PST) Date: Mon, 2 Feb 2015 13:23:45 -0700 (MST) From: ankits To: dev@spark.apache.org Message-ID: <1422908625927-10388.post@n3.nabble.com> In-Reply-To: <54CC51F1.3030604@gmail.com> References: <1422670555626-10366.post@n3.nabble.com> <54CC51F1.3030604@gmail.com> Subject: Re: Get size of rdd in memory MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Thanks for your response. So AFAICT calling parallelize(1 to1024).map(i =>KV(i, i.toString)).toSchemaRDD.cache().count(), will allow me to see the size of the schemardd in memory and parallelize(1 to1024).map(i =>KV(i, i.toString)).cache().count() will show me the size of a regular rdd. But this will not show us the size when using cacheTable() right? Like if i call parallelize(1 to1024).map(i =>KV(i, i.toString)).toSchemaRDD.registerTempTable("test") sqc.cacheTable("test") sqc.sql("SELECT COUNT(*) FROM test") the web UI does not show us the size of the cached table. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Get-size-of-rdd-in-memory-tp10366p10388.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org For additional commands, e-mail: dev-help@spark.apache.org