Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 98FDE18C25 for ; Fri, 20 Nov 2015 11:52:53 +0000 (UTC) Received: (qmail 24565 invoked by uid 500); 20 Nov 2015 11:52:52 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 24483 invoked by uid 500); 20 Nov 2015 11:52:52 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 24455 invoked by uid 99); 20 Nov 2015 11:52:52 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Nov 2015 11:52:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 8F1701A2340 for ; Fri, 20 Nov 2015 11:52:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.902 X-Spam-Level: *** X-Spam-Status: No, score=3.902 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=3, MIME_QP_LONG_LINE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id y2fCCGA4vRTh for ; Fri, 20 Nov 2015 11:52:38 +0000 (UTC) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id F226B42B7B for ; Fri, 20 Nov 2015 11:52:37 +0000 (UTC) Received: by wmec201 with SMTP id c201so17869917wme.1 for ; Fri, 20 Nov 2015 03:52:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:mime-version:subject :message-id:date:references:in-reply-to:to; bh=3ueosWfnPs7/5il2iOYOl4cjXe5hPFUVD2IO7RG7I6g=; b=koy0O7vskc1BV13u26T59aFSUe1I7kliVHfh/oYg5iU3p+psa9K3es4VtQ0bLXQbNN 9T0/cHRKR5g8V//V1LEInmj7ZztN922na0+RgxD0qkkRdEs20vu7gddTYW08Q2YREO4P t7AGUzBnq7OyF65rkeYn26lFerZ13mvLJYRxkrwlbdZNWcGxEFVmuoa8zTsKoQr5icFq Jq8Aa9aDch6hS46cVS1RMzr6DKcGmArC8FR3Kax4At4EljGtRv70RCZAgKGHF8CEZgUU BDjXe5q6qf4cdNgzSISYuTJSuo9nRyvZotwfNGZgEfI2MrBhHF87DnGfjtM4sYWr886b 7+SQ== X-Received: by 10.28.126.197 with SMTP id z188mr2042331wmc.88.1448020357167; Fri, 20 Nov 2015 03:52:37 -0800 (PST) Received: from [10.214.161.73] ([176.4.98.161]) by smtp.gmail.com with ESMTPSA id lf10sm12253836wjb.23.2015.11.20.03.52.35 for (version=TLSv1/SSLv3 cipher=OTHER); Fri, 20 Nov 2015 03:52:35 -0800 (PST) From: =?utf-8?Q?J=C3=B6rn_Franke?= Content-Type: multipart/alternative; boundary=Apple-Mail-B1194972-FC55-4486-AB58-F5E6787DDECB Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Subject: Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu Message-Id: <718ECA29-A16F-4643-B68A-67470510F6B4@gmail.com> Date: Fri, 20 Nov 2015 12:52:33 +0100 References: <01b801d12378$a0bd4020$e237c060$@peridale.co.uk> In-Reply-To: <01b801d12378$a0bd4020$e237c060$@peridale.co.uk> To: user@hive.apache.org X-Mailer: iPhone Mail (13B143) --Apple-Mail-B1194972-FC55-4486-AB58-F5E6787DDECB Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I recommend to use a Hadoop distribution containing these technologies. I th= ink you get also other useful tools for your scenario, such as Auditing usin= g sentry or ranger. > On 20 Nov 2015, at 10:48, Mich Talebzadeh wrote: >=20 > Well > =20 > =E2=80=9CI'm planning to deploy Hive on Spark but I can't find the install= ation steps. I tried to read the official '[Hive on Spark][1]' guide but it h= as problems. As an example it says under 'Configuring Yarn' `yarn.resourcema= nager.scheduler.class=3Dorg.apache.hadoop.yarn.server.resourcemanager.schedu= ler.fair.FairScheduler` but does not imply where should I do it. Also as per= the guide configurations are set in the Hive runtime shell which is not per= manent according to my knowledge.=E2=80=9D > =20 > You can do that in yarn-site.xml file which is normally under $HADOOP_HOME= /etc/hadoop. > =20 > =20 > HTH > =20 > =20 > =20 > Mich Talebzadeh > =20 > Sybase ASE 15 Gold Medal Award 2008 > A Winning Strategy: Running the most Critical Financial Data on ASE 15 > http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-09190= 8.pdf > Author of the books "A Practitioner=E2=80=99s Guide to Upgrading to Sybase= ASE 15", ISBN 978-0-9563693-0-7. > co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759= 693-0-4 > Publications due shortly: > Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-956369= 3-3-8 > Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume= one out shortly > =20 > http://talebzadehmich.wordpress.com > =20 > NOTE: The information in this email is proprietary and confidential. This m= essage is for the designated recipient only, if you are not the intended rec= ipient, you should destroy it immediately. Any information in this message s= hall not be understood as given or endorsed by Peridale Technology Ltd, its s= ubsidiaries or their employees, unless expressly so stated. It is the respon= sibility of the recipient to ensure that this email is virus free, therefore= neither Peridale Ltd, its subsidiaries nor their employees accept any respo= nsibility. > =20 > From: Dasun Hegoda [mailto:dasunhegoda@gmail.com]=20 > Sent: 20 November 2015 09:36 > To: user@hive.apache.org > Subject: Hive on Spark - Hadoop 2 - Installation - Ubuntu > =20 > Hi, > =20 > What I'm planning to do is develop a reporting platform using existing dat= a. I have an existing RDBMS which has large number of records. So I'm using.= (http://stackoverflow.com/questions/33635234/hadoop-2-7-spark-hive-jasperre= ports-scoop-architecuture) > =20 > - Scoop - Extract data from RDBMS to Hadoop > - Hadoop - Storage platform -> *Deployment Completed* > - Hive - Datawarehouse > - Spark - Read time processing -> *Deployment Completed* > =20 > I'm planning to deploy Hive on Spark but I can't find the installation ste= ps. I tried to read the official '[Hive on Spark][1]' guide but it has probl= ems. As an example it says under 'Configuring Yarn' `yarn.resourcemanager.sc= heduler.class=3Dorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair= .FairScheduler` but does not imply where should I do it. Also as per the gui= de configurations are set in the Hive runtime shell which is not permanent a= ccording to my knowledge. > =20 > Given that I read [this][2] but it does not have any steps. > =20 > Please provide me the steps to run Hive on Spark on Ubuntu as a production= system? > =20 > =20 > [1]: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+G= etting+Started > [2]: http://stackoverflow.com/questions/26018306/how-to-configure-hive-t= o-use-spark > =20 > -- > Regards, > Dasun Hegoda, Software Engineer =20 > www.dasunhegoda.com | dasunhegoda@gmail.com --Apple-Mail-B1194972-FC55-4486-AB58-F5E6787DDECB Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
I recommend to use a Hadoop distributi= on containing these technologies. I think you get also other useful tools fo= r your scenario, such as Auditing using sentry or ranger.

= On 20 Nov 2015, at 10:48, Mich Talebzadeh <mich@peridale.co.uk> wrote:

Well

 

=E2=80=9CI'm planning to deploy Hive o= n Spark but I can't find the installation steps. I tried to read the officia= l '[Hive on Spark][1]' guide but it has problems. As an example it says unde= r 'Configuring Yarn' `yarn.resourcemanager.scheduler.class=3Dorg.apache.hado= op.yarn.server.resourcemanager.scheduler.fair.FairScheduler` but does not im= ply where should I do it. Also as per the guide configurations are set in th= e Hive runtime shell which is not permanent according to my knowledge.=E2=80= =9D

 

You can do that in yarn-site.xml file which is normally under $HADO= OP_HOME/etc/hadoop.

 

 

HTH

 =

 =

 =

Mich Talebzadeh

 

Sybase ASE= 15 Gold Medal Award 2008

A Winning Strategy: Running the most= Critical Financial Data on ASE 15

= http://login.sybase.com/files/Product_Overviews/ASE-Winning-= Strategy-091908.pdf

Author of the books "A Practitioner=E2=80=99= s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.

= co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-975= 9693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous E= nvironments, ISBN: 978-0-9563693-3-8<= /p>

Oracle and Sybase, C= oncepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidentia= l. This message is for the designated recipient only, if you are not the int= ended recipient, you should destroy it immediately. Any information in this m= essage shall not be understood as given or endorsed by Peridale Technology L= td, its subsidiaries or their employees, unless expressly so stated. It is t= he responsibility of the recipient to ensure that this email is virus free, t= herefore neither Peridale Ltd, its subsidiaries nor their employees accept a= ny responsibility.

 

From: Dasun Hegoda [mailto:dasunhegoda@gmail.com]
= Sent: 20 November 2015 09:36
To: user@hive.apache.org
Subject: Hive on Spark - Hadoo= p 2 - Installation - Ubuntu

 

Hi,

 

W= hat I'm planning to do is develop a reporting platform using existing data. I= have an existing RDBMS which has large number of records. So I'm using. (http://stackoverflow.com/questions/3363523= 4/hadoop-2-7-spark-hive-jasperreports-scoop-architecuture)

 

 - Scoop - Extract data from RDBMS to Hadoop

=

 - Hadoop - Storage platform -> *D= eployment Completed*

 -= Hive - Datawarehouse

 = - Spark - Read time processing -> *Deployment Completed*

 

I'm planning to deploy Hive on Spark but I can't find the install= ation steps. I tried to read the official '[Hive on Spark][1]' guide but it h= as problems. As an example it says under 'Configuring Yarn' `yarn.resourcema= nager.scheduler.class=3Dorg.apache.hadoop.yarn.server.resourcemanager.schedu= ler.fair.FairScheduler` but does not imply where should I do it. Also as per= the guide configurations are set in the Hive runtime shell which is not per= manent according to my knowledge.

 

Given that I re= ad [this][2] but it does not have any steps.

 

Plea= se provide me the steps to run Hive on Spark on Ubuntu as a production syste= m?

--

Regards,

Dasun Hegoda, Software E= ngineer  
www.dasunhegoda.com | dasunhegoda@gmail.com=

<= /body>= --Apple-Mail-B1194972-FC55-4486-AB58-F5E6787DDECB--