Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 044F118F24 for ; Mon, 23 Nov 2015 07:04:54 +0000 (UTC) Received: (qmail 60388 invoked by uid 500); 23 Nov 2015 07:04:52 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 60319 invoked by uid 500); 23 Nov 2015 07:04:52 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 60309 invoked by uid 99); 23 Nov 2015 07:04:52 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2015 07:04:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 165A418099D for ; Mon, 23 Nov 2015 07:04:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.911 X-Spam-Level: ** X-Spam-Status: No, score=2.911 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 2Agg41JiwrJy for ; Mon, 23 Nov 2015 07:04:37 +0000 (UTC) Received: from mail-yk0-f181.google.com (mail-yk0-f181.google.com [209.85.160.181]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 1D7F4206E3 for ; Mon, 23 Nov 2015 07:04:36 +0000 (UTC) Received: by ykdv3 with SMTP id v3so227771349ykd.0 for ; Sun, 22 Nov 2015 23:04:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ED2zZsTIbJBfZVVMS+ktVHIPi4Ks1TuEu4T12WfOJlc=; b=IGBuMhcx/tBjA03CuLojT4/VBoxORfdZqnxMuziudVJninDmQ4BmQenYsoClr/AH8b rImsHWjwhtLh3cBZNT4047i/o7XKIx+liNW/9R/A7kv6lKhGejFmSmTTGMp5DlcSboit NEGedkJa1VHtbNUv2vqGdXWO4gNwlB0cqIatc3mucyyMElwV83oLOSvyqQD5NiffVJor Ju7nZ3I40vRV7dMXl9rN5o0bRanUrpqccNGSD7Wlgs6UQDSuygQup4+YCjwVwd3Fm9zJ fmwBU+V6HsofdzwJKMsqNWL1JWpmMUnSz+j/8HKdgIzuQbZdFRBsN2gUdsvfxDSAG2uN SkQQ== MIME-Version: 1.0 X-Received: by 10.13.255.4 with SMTP id p4mr22238266ywf.54.1448262275027; Sun, 22 Nov 2015 23:04:35 -0800 (PST) Received: by 10.13.228.67 with HTTP; Sun, 22 Nov 2015 23:04:34 -0800 (PST) In-Reply-To: References: <01b801d12378$a0bd4020$e237c060$@peridale.co.uk> <718ECA29-A16F-4643-B68A-67470510F6B4@gmail.com> Date: Mon, 23 Nov 2015 12:34:34 +0530 Message-ID: Subject: Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu From: Dasun Hegoda To: user@hive.apache.org Content-Type: multipart/alternative; boundary=94eb2c0874e671deca05252fd63b --94eb2c0874e671deca05252fd63b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Anyone???? On Sat, Nov 21, 2015 at 1:32 PM, Dasun Hegoda wrote= : > Thank you very much but I would like to do the integration of these > components myself rather than using a packaged distribution. I think I ha= ve > come to right place. Can you please kindly tell me the configuration > steps run Hive on Spark? > > At least someone please elaborate these steps. > > https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting= +Started > . > > Because at the latter part of the guide configurations are set in the > Hive runtime shell which is not permanent according to my knowledge. > > Please help me to get this done. Also I'm planning write a detailed guide > with configuration steps to run Hive on Spark. So others can benefited fr= om > it and not troubled like me. > > Can someone please kindly tell me the configuration steps run Hive on > Spark? > > > On Sat, Nov 21, 2015 at 12:28 PM, Sai Gopalakrishnan < > sai.gopalakrishnan@aspiresys.com> wrote: > >> Hi everyone, >> >> >> Thank you for your responses. I think Mich's suggestion is a great one, >> will go with it. As Alan suggested, using compactor in Hive should help = out >> with managing the delta files. >> >> >> @Dasun, pardon me for deviating from the topic. Regarding configuration, >> you could try a packaged distribution (Hortonworks , Cloudera or MapR) >> like J=C3=B6rn Franke said. I use Hortonworks, its open-source and comp= atible >> with Linux and Windows, provides detailed documentation for installation >> and can be installed in less than a day provided you're all set with the >> hardware. http://hortonworks.com/hdp/downloads/ >> >> Download Hadoop - Hortonworks >> Download Apache Hadoop for the enterprise with Hortonworks Data Platform= . >> Data access, storage, governance, security and operations across Linux a= nd >> Windows >> Read more... >> >> >> Regards, >> >> Sai >> >> ------------------------------ >> *From:* Dasun Hegoda >> *Sent:* Saturday, November 21, 2015 8:00 AM >> *To:* user@hive.apache.org >> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu >> >> Hi Mich, Hi Sai, Hi Jorn, >> >> Thank you very much for the information. I think we are deviating from >> the original question. Hive on Spark on Ubuntu. Can you please kindly te= ll >> me the configuration steps? >> >> >> >> On Fri, Nov 20, 2015 at 11:10 PM, J=C3=B6rn Franke >> wrote: >> >>> I think the most recent versions of cloudera or Hortonworks should >>> include all these components - try their Sandboxes. >>> >>> On 20 Nov 2015, at 12:54, Dasun Hegoda wrote: >>> >>> Where can I get a Hadoop distribution containing these technologies? >>> Link? >>> >>> On Fri, Nov 20, 2015 at 5:22 PM, J=C3=B6rn Franke >>> wrote: >>> >>>> I recommend to use a Hadoop distribution containing these technologies= . >>>> I think you get also other useful tools for your scenario, such as Aud= iting >>>> using sentry or ranger. >>>> >>>> On 20 Nov 2015, at 10:48, Mich Talebzadeh wrote: >>>> >>>> Well >>>> >>>> >>>> >>>> =E2=80=9CI'm planning to deploy Hive on Spark but I can't find the >>>> installation steps. I tried to read the official '[Hive on Spark][1]' = guide >>>> but it has problems. As an example it says under 'Configuring Yarn' >>>> `yarn.resourcemanager.scheduler.class=3Dorg.apache.hadoop.yarn.server.= resourcemanager.scheduler.fair.FairScheduler` >>>> but does not imply where should I do it. Also as per the guide >>>> configurations are set in the Hive runtime shell which is not permanen= t >>>> according to my knowledge.=E2=80=9D >>>> >>>> >>>> >>>> You can do that in yarn-site.xml file which is normally under >>>> $HADOOP_HOME/etc/hadoop. >>>> >>>> >>>> >>>> >>>> >>>> HTH >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Mich Talebzadeh >>>> >>>> >>>> >>>> *Sybase ASE 15 Gold Medal Award 2008* >>>> >>>> A Winning Strategy: Running the most Critical Financial Data on ASE 15 >>>> >>>> >>>> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-0= 91908.pdf >>>> >>>> Author of the books* "A Practitioner=E2=80=99s Guide to Upgrading to S= ybase >>>> ASE 15", ISBN 978-0-9563693-0-7*. >>>> >>>> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >>>> 978-0-9759693-0-4* >>>> >>>> *Publications due shortly:* >>>> >>>> *Complex Event Processing in Heterogeneous Environments*, ISBN: >>>> 978-0-9563693-3-8 >>>> >>>> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, = volume >>>> one out shortly >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> >>>> NOTE: The information in this email is proprietary and confidential. >>>> This message is for the designated recipient only, if you are not the >>>> intended recipient, you should destroy it immediately. Any information= in >>>> this message shall not be understood as given or endorsed by Peridale >>>> Technology Ltd, its subsidiaries or their employees, unless expressly = so >>>> stated. It is the responsibility of the recipient to ensure that this = email >>>> is virus free, therefore neither Peridale Ltd, its subsidiaries nor th= eir >>>> employees accept any responsibility. >>>> >>>> >>>> >>>> *From:* Dasun Hegoda [mailto:dasunhegoda@gmail.com >>>> ] >>>> *Sent:* 20 November 2015 09:36 >>>> *To:* user@hive.apache.org >>>> *Subject:* Hive on Spark - Hadoop 2 - Installation - Ubuntu >>>> >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> What I'm planning to do is develop a reporting platform using existing >>>> data. I have an existing RDBMS which has large number of records. So I= 'm >>>> using. ( >>>> http://stackoverflow.com/questions/33635234/hadoop-2-7-spark-hive-jasp= erreports-scoop-architecuture >>>> ) >>>> >>>> >>>> >>>> - Scoop - Extract data from RDBMS to Hadoop >>>> >>>> - Hadoop - Storage platform -> *Deployment Completed* >>>> >>>> - Hive - Datawarehouse >>>> >>>> - Spark - Read time processing -> *Deployment Completed* >>>> >>>> >>>> >>>> I'm planning to deploy Hive on Spark but I can't find the installation >>>> steps. I tried to read the official '[Hive on Spark][1]' guide but it = has >>>> problems. As an example it says under 'Configuring Yarn' >>>> `yarn.resourcemanager.scheduler.class=3Dorg.apache.hadoop.yarn.server.= resourcemanager.scheduler.fair.FairScheduler` >>>> but does not imply where should I do it. Also as per the guide >>>> configurations are set in the Hive runtime shell which is not permanen= t >>>> according to my knowledge. >>>> >>>> >>>> >>>> Given that I read [this][2] but it does not have any steps. >>>> >>>> >>>> >>>> Please provide me the steps to run Hive on Spark on Ubuntu as a >>>> production system? >>>> >>>> >>>> >>>> >>>> >>>> [1]: >>>> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Gett= ing+Started >>>> >>>> [2]: >>>> http://stackoverflow.com/questions/26018306/how-to-configure-hive-to-u= se-spark >>>> >>>> >>>> >>>> -- >>>> >>>> Regards, >>>> >>>> Dasun Hegoda, Software Engineer >>>> www.dasunhegoda.com | dasunhegoda@gmail.com >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Dasun Hegoda, Software Engineer >>> www.dasunhegoda.com | dasunhegoda@gmail.com >>> >>> >> >> >> -- >> Regards, >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunhegoda@gmail.com >> [image: Aspire Systems] >> >> This e-mail message and any attachments are for the sole use of the >> intended recipient(s) and may contain proprietary, confidential, trade >> secret or privileged information. Any unauthorized review, use, disclosu= re >> or distribution is prohibited and may be a violation of law. If you are = not >> the intended recipient, please contact the sender by reply e-mail and >> destroy all copies of the original message. >> > > > > -- > Regards, > Dasun Hegoda, Software Engineer > www.dasunhegoda.com | dasunhegoda@gmail.com > --=20 Regards, Dasun Hegoda, Software Engineer www.dasunhegoda.com | dasunhegoda@gmail.com --94eb2c0874e671deca05252fd63b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Anyone????

On Sat, Nov 21, 2015 at 1:32 PM, Dasun Hegoda <dasunh= egoda@gmail.com> wrote:
Thank you very much but I would like to do the integration of= these components myself rather than using a packaged distribution. I think= I have come to right place.=C2=A0Can you = please kindly tell me the configuration steps run Hive on Spark?
At least someone please elaborate these steps.
ht= tps://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Sta= rted.

=
Because at the latter part of the guide=C2=A0configurations are set in the Hive runtime shell which is not= permanent according to my knowledge.

Please = help me to get this done. Also I'm planning write a detailed guide with= =C2=A0configuration steps to run Hive on Spark. So others can=C2=A0benefite= d=C2=A0from it and not=C2=A0troubled=C2=A0like me.

Can someone please kindly tell me the configuration steps run Hive on= Spark?


On= Sat, Nov 21, 2015 at 12:28 PM, Sai Gopalakrishnan <sai.gop= alakrishnan@aspiresys.com> wrote:

Hi everyone,


Thank you for your responses. I think Mich's suggestion is a great o= ne, will go with it. As Alan suggested, using compactor in Hive should help= out with managing the delta files.


@Dasun, pardon me for deviating from the topic. Regarding configuration,= you could try a packaged distribution (Hortonworks ,=C2=A0Cloudera or MapR= ) like=C2=A0=C2=A0J=C3=B6rn Franke said. I use Hortonworks, its open-source= and compatible with Linux and Windows, provides detailed documentation for installation and can be installed in less than a day pro= vided you're all set with the hardware.=C2=A0http://hortonworks.com/hdp/downlo= ads/=C2=A0
Download Hadoop - Hortonworks
Download Apache Hadoop for the enterprise with Hortonworks Data Platform. D= ata access, storage, governance, security and operations across Linux and W= indows


Regards,

Sai


From: Dasun Hegoda <dasunhegoda@gmail.com>
Sent: Saturday, November 21, 2015 8:00 AM
To: user@h= ive.apache.org
Subject: Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu
=C2=A0
Hi Mich, Hi Sai, Hi Jorn,

Thank you very much for the information. I think we are deviating from the = original question. Hive on Spark on Ubuntu. Can you please kindly tell me t= he configuration steps?



On Fri, Nov 20, 2015 at 11:10 PM, J=C3=B6rn Fran= ke <jornfranke@gm= ail.com> wrote:
I think the most recent versions of cloudera or Hortonworks should inc= lude all these components - try their Sandboxes.=C2=A0

On 20 Nov 2015, at 12:54, Dasun Hegoda <dasunhegoda@gmail.c= om> wrote:

Where can I get=C2=A0a Ha= doop distribution containing these technologies? Link?

On Fri, Nov 20, 2015 at 5:22 PM, J=C3=B6rn Frank= e <jornfranke@gm= ail.com> wrote:
I recommend to use a Hadoop distribution containing these technologies= . I think you get also other useful tools for your scenario, such as Auditi= ng using sentry or ranger.

On 20 Nov 2015, at 10:48, Mich Talebzadeh <mich@peridale.co.uk> wrote:

Well

=C2=A0

=E2=80=9CI'm planning to deploy Hive on Sp= ark but I can't find the installation steps. I tried to read the offici= al '[Hive on Spark][1]' guide but it has problems. As an example it says under 'Configuring Yarn' `yarn.resourcemanager.scheduler.c= lass=3Dorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSch= eduler` but does not imply where should I do it. Also as per the guide conf= igurations are set in the Hive runtime shell which is not permanent according to my knowledge.=E2=80=9D

=C2=A0

You can do that in yarn-site.xml file which is norma= lly under $HADOOP_HOME/etc/hadoop.

=C2=A0

=C2=A0

HTH

=C2=A0

=C2=A0

=C2=A0

Mich Talebzadeh

=C2=A0

Sybase ASE 15 Gold Medal Award 2008<= /span>

A Winning= Strategy: Running the most Critical Financial Data on ASE 15=

http://login.sybase= .com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf<= /u>

Author of th= e books "A Practitioner=E2=80=99s Guide to Upgrading to Sybase ASE = 15", ISBN 978-0-9563693-0-7.

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-97= 59693-0-4

Publications due shortly:=

Complex E= vent Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Co= ncepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

=C2= =A0

http://talebzadehmich.wordpress.= com

=C2=A0

NOTE: The information in this email is pro= prietary and confidential. This message is for the designated recipient onl= y, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not b= e understood as given or endorsed by Peridale Technology Ltd, its subsidiar= ies or their employees, unless expressly so stated. It is the responsibilit= y of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries= nor their employees accept any responsibility.<= u>

=C2=A0

From: = Dasun Hegoda [ma= ilto:dasunhegoda@gmail.com]
Sent: 20 November 2015 09:36
To: user@h= ive.apache.org
Subject: Hive on Spark - Hadoop 2 - Installation - Ubuntu<= /u>

=C2=A0

Hi,

=C2=A0

What I'm planning to do is develop a reporting p= latform using existing data. I have an existing RDBMS which has large numbe= r of records. So I'm using. (http://stackoverflow.com/questions/33635234/hadoop-2-7-spark-h= ive-jasperreports-scoop-architecuture)

=C2=A0

=C2=A0- Scoop - Extract data from RDBMS to Hadoop=

=C2=A0- Hadoop - Storage platform -> *Deployment = Completed*

=C2=A0- Hive - Datawarehouse

=C2=A0- Spark - Read time processing -> *Deployme= nt Completed*

=C2=A0

I'm planning to deploy Hive on Spark but I can&#= 39;t find the installation steps. I tried to read the official '[Hive o= n Spark][1]' guide but it has problems. As an example it says under = 9;Configuring Yarn' `yarn.resourcemanager.scheduler.class=3Dorg.apache.= hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` but does not imply where should I do it. Also as per the guide configurati= ons are set in the Hive runtime shell which is not permanent according to m= y knowledge.

=C2=A0

Given that I read [this][2] but it does not have any= steps.

=C2=A0

Please provide me the steps to run Hive on Spark on = Ubuntu as a production system?

=C2=A0

=C2=A0

=C2=A0

--

Regards,=

Dasun Hegoda= , Software Engineer=C2=A0=C2=A0
www.dasunh= egoda.com=C2=A0| dasunhegoda@gmai= l.com




--
Regards,
Dasun Hegoda, Software Engineer=C2=A0=C2=A0
www.dasunhegoda.c= om=C2=A0| dasunhegoda@gmai= l.com



--
Regards,
Dasun Hegoda, Software Engineer=C2=A0=C2=A0
www.dasunhegoda.c= om=C2=A0| dasunhegoda@gmai= l.com
3D"Aspire

This e-mail message and any attachments are for the sole use of the intende= d recipient(s) and may contain proprietary, confidential, trade secret or p= rivileged information. Any unauthorized review, use, disclosure or distribu= tion is prohibited and may be a violation of law. If you are not the intended recipient, please contact th= e sender by reply e-mail and destroy all copies of the original message.




--
<= font color=3D"#888888">Regards,
Dasun He= goda, Software Engineer=C2=A0= =C2=A0
www.dasunhegoda.com=C2=A0| dasunhegoda@gmail.com



--
=
Regards,
Dasun Hegoda, Software Engineer=C2=A0=C2=A0
www.= dasunhegoda.com=C2=A0| dasunhegoda@gmail.com
--94eb2c0874e671deca05252fd63b--