Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8E88211F17 for ; Thu, 28 Aug 2014 19:45:51 +0000 (UTC) Received: (qmail 97529 invoked by uid 500); 28 Aug 2014 19:45:49 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 97471 invoked by uid 500); 28 Aug 2014 19:45:49 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 97458 invoked by uid 99); 28 Aug 2014 19:45:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Aug 2014 19:45:49 +0000 X-ASF-Spam-Status: No, hits=2.3 required=5.0 tests=SPF_SOFTFAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of skrishna.id@gmail.com does not designate 216.139.236.26 as permitted sender) Received: from [216.139.236.26] (HELO sam.nabble.com) (216.139.236.26) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Aug 2014 19:45:23 +0000 Received: from ben.nabble.com ([192.168.236.152]) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1XN5dK-0001M0-Hr for user@spark.incubator.apache.org; Thu, 28 Aug 2014 12:45:22 -0700 Date: Thu, 28 Aug 2014 12:45:22 -0700 (PDT) From: SK To: user@spark.incubator.apache.org Message-ID: <1409255122546-13056.post@n3.nabble.com> In-Reply-To: <609734400.2074140.1409085236839.JavaMail.zimbra@stanford.edu> References: <1409081880360-12847.post@n3.nabble.com> <609734400.2074140.1409085236839.JavaMail.zimbra@stanford.edu> Subject: Re: OutofMemoryError when generating output MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, Thanks for the response. I tried to use countByKey. But I am not able to write the output to console or to a file. Neither collect() nor saveAsTextFile() work for the Map object that is generated after countByKey(). val x = sc.textFile(baseFile)).map { line => val fields = line.split("\t") (fields(11), fields(6)) // extract (month, user_id) }.distinct().countByKey() x.saveAsTextFile(...) // does not work. generates an error that saveAstextFile is not defined for Map object Is there a way to convert the Map object to an object that I can output to console and to a file? thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/OutofMemoryError-when-generating-output-tp12847p13056.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org