Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C23818FC8 for ; Thu, 28 May 2015 00:45:42 +0000 (UTC) Received: (qmail 69493 invoked by uid 500); 28 May 2015 00:45:39 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 69406 invoked by uid 500); 28 May 2015 00:45:39 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 69396 invoked by uid 99); 28 May 2015 00:45:39 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 May 2015 00:45:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D75E41A363F for ; Thu, 28 May 2015 00:45:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.888 X-Spam-Level: ** X-Spam-Status: No, score=2.888 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 4gqi2IyJjzTT for ; Thu, 28 May 2015 00:45:32 +0000 (UTC) Received: from nm3-vm4.bullet.mail.gq1.yahoo.com (nm3-vm4.bullet.mail.gq1.yahoo.com [98.136.218.147]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 59BD320A93 for ; Thu, 28 May 2015 00:45:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1432773866; bh=o2sNJv+VuHaVaFHIH6wlWzIwJV1XKHMrocXGVoqxwPs=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=K628Bdtjti+25Jdlez3ym8m5V/zpSIi/QBGxidEXVupQlx1lvcFZh0+WIQTTTNcv0AEShcBY5+8gPRX4BeRX5Bljj+aWnkX0/KKm9uLswMGqqAbiZIXNHFq7Z6TG6CrY7Y7zLhlnnZmgIYe0Obyun+2eOQ02TqbhdTiPLgPQpBC9LAaIFCT9JjxZ6kXFux1WSCrQvOyBoMFBhszX27zttF5S1IfLOJBqgfQwtLzmLWq+D8eE3viq2QQHaYO6cNJXAfBDwTORDGRZW0XCC8DfxtAXYpJadS/ELhSMvHKClVbdN6pu5rDTucdQeNSUAxzxTIMluJP9WeoYGIpvQxnPyw== Received: from [216.39.60.181] by nm3.bullet.mail.gq1.yahoo.com with NNFMP; 28 May 2015 00:44:26 -0000 Received: from [98.137.12.229] by tm17.bullet.mail.gq1.yahoo.com with NNFMP; 28 May 2015 00:44:26 -0000 Received: from [127.0.0.1] by omp1037.mail.gq1.yahoo.com with NNFMP; 28 May 2015 00:44:26 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 431395.6905.bm@omp1037.mail.gq1.yahoo.com X-YMail-OSG: dg6xHZkVM1msCRm2lHqH1MLG7XQU6df09dgP6TY6vMsc5W6MiG49EuHtE9kSgSk kgoz5IXZVfIaUtPW3Gjl7IyeGAy.F3DXCSngh2qzWnm5kI3ALROZmMEvL.woNYAC7nUr016IJB5t PblLAX1cMdZKDd72S5hbpJA0OqbtIi_4rcBNHx8k3eq9CiazxT.ygD_AXGrSmC8VDY1ZIiwSrKt9 Duxiffu2k4.VoXxG.j2_0CgM6idUF9aNxqSRTUaUsMswu71GBtwEJsuvPXyjkQInoUrO2mUdLS0D hdhu3xvJlRgPjedEKCU9tXVSmo3sZY1njb6xBr5Lwl09ztuRPnu8Em.mqRCX8gWsEEMXb0iw8t3Y VT96L6f2sZNfwykyCGK8cCWyf5lXmEyFjUULvQ.ubRi8GGLPjqxCPRKloI8tKsTeHQATlS99MKHn elG9bkOQdC35ZvYp70PyA0tkMotnSh6m_Rh0nG0_Z3Q3YAq01e_J6LsCihQcaeo3EO1DihWeRy7A MnzHe9cmxzw-- Received: by 98.137.12.54; Thu, 28 May 2015 00:44:25 +0000 Date: Thu, 28 May 2015 00:44:25 +0000 (UTC) From: Sanjay Subramanian Reply-To: Sanjay Subramanian To: "user@hive.apache.org" Message-ID: <536502230.31542.1432773865201.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Pointing SparkSQL to existing Hive Metadata with data file locations in HDFS MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_31541_387836146.1432773865198" ------=_Part_31541_387836146.1432773865198 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable hey guys On the Hive/Hadoop ecosystem we have using Cloudera distribution CDH 5.2.x = , there are about 300+ hive tables.The data is stored an text (moving slowl= y to Parquet) on HDFS.I want to use SparkSQL and point to the Hive metadata= and be able to define JOINS etc using a programming structure like this=C2= =A0 import org.apache.spark.sql.hive.HiveContextval sqlContext =3D new HiveCont= ext(sc)val schemaRdd =3D sqlContext.sql("some complex SQL") Is that the way to go ? Some guidance will be great. thanks sanjay ------=_Part_31541_387836146.1432773865198 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
hey guys

On the Hive/Hadoop ecosystem= we have using Cloudera distribution CDH 5.2.x , there are about 300+ hive = tables.
The data is stored= an text (moving slowly to Parquet) on HDFS.
I want to use SparkSQL and point to the Hive= metadata and be able to define JOINS etc using a programming structure lik= e this 

import org.apache.spark.sql.hive.HiveCont= ext
val sqlContext =3D new HiveContext(sc)
val schemaRdd = =3D sqlContext.sql("some complex SQL")


Is that the way to = go ? Some guidance will be great.

thanks

sanjay


=

= ------=_Part_31541_387836146.1432773865198--