Return-Path: 
 <user-return-44011-apmail-spark-user-archive=spark.apache.org@spark.apache.org>
X-Original-To: apmail-spark-user-archive@minotaur.apache.org
Delivered-To: apmail-spark-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8E2418093
	for <apmail-spark-user-archive@minotaur.apache.org>;
 Fri,  9 Oct 2015 23:55:26 +0000 (UTC)
Received: (qmail 46157 invoked by uid 500); 9 Oct 2015 23:55:22 -0000
Delivered-To: apmail-spark-user-archive@spark.apache.org
Received: (qmail 46066 invoked by uid 500); 9 Oct 2015 23:55:22 -0000
Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@spark.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@spark.apache.org>
List-Post: <mailto:user@spark.apache.org>
List-Id: <user.spark.apache.org>
Delivered-To: mailing list user@spark.apache.org
Received: (qmail 46055 invoked by uid 99); 9 Oct 2015 23:55:22 -0000
Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2015 23:55:22 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org)
 with ESMTP id A4129C0F5F
	for <user@spark.apache.org>; Fri,  9 Oct 2015 23:55:21 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 3.487
X-Spam-Level: ***
X-Spam-Status: No, score=3.487 tagged_above=-999 required=6.31
	tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2,
	SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001, URI_HEX=1.313]
	autolearn=disabled
Received: from mx1-eu-west.apache.org ([10.40.0.8])
	by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new,
 port 10024)
	with ESMTP id c7gpFrmZ6Wx0 for <user@spark.apache.org>;
	Fri,  9 Oct 2015 23:55:12 +0000 (UTC)
Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43])
	by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with
 ESMTP id DE7AB206E8
	for <user@spark.apache.org>; Fri,  9 Oct 2015 23:55:11 +0000 (UTC)
Received: from mben.nabble.com (unknown [162.253.133.72])
	by mwork.nabble.com (Postfix) with ESMTP id 2393B2ACC0EF
	for <user@spark.apache.org>; Fri,  9 Oct 2015 16:55:58 -0700 (PDT)
Date: Fri, 9 Oct 2015 16:55:11 -0700 (MST)
From: Abhisheks <smartshobhu@gmail.com>
To: user@spark.apache.org
Message-ID: <1444434911224-25005.post@n3.nabble.com>
Subject: SQLcontext changing String field to Long
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi there,

I have saved my records in to parquet format and am using Spark1.5. But when
I try to fetch the columns it throws exception*
java.lang.ClassCastException: java.lang.Long cannot be cast to
org.apache.spark.unsafe.types.UTF8String*.

This filed is saved as String while writing parquet. so here is the sample
code and output for the same..

logger.info("troubling thing is ::" +
sqlContext.sql(fileSelectQuery).schema().toString());
DataFrame df= sqlContext.sql(fileSelectQuery);
JavaRDD<Row> rdd2 = df.toJavaRDD();

First Line in the code (Logger) prints this: 
troubling thing is ::StructType(StructField(batch_id,StringType,true))

But the moment after it the execption comes up. 

Any idea why it is treating the filed as Long? (yeah one unique thing about
column is it is always a number e.g. Time-stamp).

Any help is appreciated.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SQLcontext-changing-String-field-to-Long-tp25005.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org