Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0DFB518EE1 for ; Fri, 4 Mar 2016 05:39:06 +0000 (UTC) Received: (qmail 59289 invoked by uid 500); 4 Mar 2016 05:39:01 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 59173 invoked by uid 500); 4 Mar 2016 05:39:01 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 59162 invoked by uid 99); 4 Mar 2016 05:39:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2016 05:39:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0075BC26E4 for ; Fri, 4 Mar 2016 05:39:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id z0Ej9p4yO9HZ for ; Fri, 4 Mar 2016 05:39:00 +0000 (UTC) Received: from mail-oi0-f48.google.com (mail-oi0-f48.google.com [209.85.218.48]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E38365F56E for ; Fri, 4 Mar 2016 05:38:59 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id m82so31315337oif.1 for ; Thu, 03 Mar 2016 21:38:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=SlA7Mxq1iOkBcx1uNt9+H1sMcfuREhtCD3IzJJ/5gwc=; b=XgjUE3qgPzV1FB1P5rOOmO55RUWpLnaegUdSsgsXCiIBkU6i0h30WEDnwakYqa28cm /o30H+h9nZN3XzSn/GLV3ewyRx6qnmLYX8jfr6uRXGAyVJBIRpjPyQEaTt+xxdUBvkA4 /OKp4BzmZ8DCR7f1nRRfbWDLUwvK0d66SrHGsjn+LfYqTotHuqMSnxK6IEzJ+kUDmlqI AZC2O3wXs19Px8YjN7H8sWMIhGQTC/kuQHqT3a4ER7mhrZwY7nzavksHpMkduaDIDEKr xOeh9QMAE6+/h0mqrZ2ygkxHrq5mhQl+qoLfbWlS7t9OgBdXX66yrXH3Rel1sOTQ0nrW TA6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=SlA7Mxq1iOkBcx1uNt9+H1sMcfuREhtCD3IzJJ/5gwc=; b=BUhR8RRxPojTJWFvtAV1Ad2d+J7dterxhWKx+krZTJ2uSGUMeHLxOB/7v5FogdBL+3 KMAvIzpK7kPMcBuUsptPDUCgelidGozAPxJPA7QyGDpytxYCtTT5Wj49O4CBh/aLT5Vu WSkeTZ6oUUkfOY4+rcsslRNcG4ZRzLAq6KIBytC8fRsk4KEX9yniWMv6R914KVbX1/n8 +QPn0UkQ6LRSFCnEx6/qNu1k4+74gizTBj89laNoDTBmSow5xN1orti1UfKrkcub6YNU DO2xOoQcIIfLzeG9yeUIdzI+KMeWC/f+kdC+H2pDkz9FsFJEMU7X+7xQO/bsPErcyd9b qMEg== X-Gm-Message-State: AD7BkJKm80bn/tYNUTqkousnYcCvYhCa/HDxCAJoyQeo/pk7dxlvnmhF4aAs8vF3agpa7bvDYkbzTDqebuayqQ== X-Received: by 10.202.77.67 with SMTP id a64mr4057401oib.123.1457069933024; Thu, 03 Mar 2016 21:38:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.185.195 with HTTP; Thu, 3 Mar 2016 21:38:13 -0800 (PST) From: Divya Gehlot Date: Fri, 4 Mar 2016 13:38:13 +0800 Message-ID: Subject: Spark 1.5.2 - Read custom schema from file To: "user @spark" , user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1134e92ac55b28052d328733 --001a1134e92ac55b28052d328733 Content-Type: text/plain; charset=UTF-8 Hi, I have defined a custom schema as shown below : val customSchema = StructType( > StructField("year", IntegerType, true), > StructField("make", StringType, true), > StructField("model", StringType, true), > StructField("comment", StringType, true), StructField("blank", StringType, true)) Is there any way instead of defining it spark job file I can read from file. I am using Spark-csv to read my data file val df = sqlContext.read .format("com.databricks.spark.csv") .option("header", "true") // Use first line of all files as header .schema(customSchema) .load("cars.csv")val selectedData = df.select("year", "model") selectedData.write .format("com.databricks.spark.csv") .option("header", "true") .save("newcars.csv") --001a1134e92ac55b28052d328733 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
I have defined a custom schema as shown below :
val customSche=
ma =3D StructType(
    StructField("year", IntegerType, true),=

    StructField("make", StringType, true),
    StructField=
("model", StringType, true),
    StructField("comment&quo=
t;, StringType, true),
    StructField("=
blank", StringType, true))

Is there an=
y way instead of defining it spark job file I can read from file.
I am using Spark-csv to read my data file

=C2=A0= val df =3D sqlContext.read
.format("com.databricks.spark.csv")
.option("header"= , "true") // Use first line of all files as header
<= /span> .schema(customSchema)
.load("cars.csv"= ) val selectedData =3D df.select("year", "model")
selectedData.write
.format("com.databricks.spark.csv")
.option("header", "true")
.save(<= span style=3D"font-size:10.4px">"newcars.csv")
--001a1134e92ac55b28052d328733--