Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F1FFD6C4 for ; Fri, 18 Jan 2013 16:39:18 +0000 (UTC) Received: (qmail 21685 invoked by uid 500); 18 Jan 2013 16:39:13 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 20979 invoked by uid 500); 18 Jan 2013 16:39:13 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 20970 invoked by uid 99); 18 Jan 2013 16:39:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2013 16:39:13 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.82.49] (HELO mail-wg0-f49.google.com) (74.125.82.49) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2013 16:39:06 +0000 Received: by mail-wg0-f49.google.com with SMTP id 15so2266795wgd.4 for ; Fri, 18 Jan 2013 08:38:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:x-originating-ip:in-reply-to:references :date:message-id:subject:from:to:content-type:x-gm-message-state; bh=3zFPDoRx9BxsVQHDgfAxhrrzRz89DWoCooG/xWuLdNI=; b=axadKzFNIgsfz90axehT9MnvpI6/sW1BuAJd/LCq6+CbtCMUIGE25ot72XYautLUBA H6+V7l9JUnOAA1oVEoIFpFV+8W+s7Rllju9+4Gw5F5zd8OO7OCRrUw5ONzFn5y0Pcorw INS/BMzqqTGXApo3j2NMQXiDSLcuSH5jA7NShkqADarwK6n3WL7hErP9s5Nb4dcSO4uV qin80rFEPUdR/i2N8/sCHEehuvQnv1rAxRX+JuyCZM0HQeRHP0JclYfWGiJPTnbth9nY 5T2DdhYnLX9c6bkrdp5MXcYH6FnPNTCxUKevoax74zcsb1aOK7xlgDG4hBteW1W80nI5 ajMg== MIME-Version: 1.0 X-Received: by 10.180.99.227 with SMTP id et3mr4745686wib.6.1358527126072; Fri, 18 Jan 2013 08:38:46 -0800 (PST) Received: by 10.195.13.198 with HTTP; Fri, 18 Jan 2013 08:38:46 -0800 (PST) X-Originating-IP: [99.140.182.24] In-Reply-To: References: <556325346CA26341B6F0530E07F90D96016C64CD96B1@GBGH-EXCH-CMS.sig.ads> <556325346CA26341B6F0530E07F90D96016C64CD96B3@GBGH-EXCH-CMS.sig.ads> Date: Fri, 18 Jan 2013 10:38:46 -0600 Message-ID: Subject: Re: HWI use on AWS/EMR From: Dean Wampler To: user@hive.apache.org Content-Type: multipart/alternative; boundary=f46d041826aac46d8104d392be89 X-Gm-Message-State: ALoCoQljHMcG8gTtW5Wl4WVkJjMHfPKj7MNx6wjMSB8pEVOWwD7ObJD60LZlPwQ/WVYQZSnUmlk5 X-Virus-Checked: Checked by ClamAV on apache.org --f46d041826aac46d8104d392be89 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yes, this is straightforward to set up in Putty. Ariel, thanks for providing the details I was too lazy to mention ;) One flaw with this approach is that only your machine will have access to HWI. That is, the port is forwarded only to your machine. Making the ports public on the master node means anyone on the interwebs can see it. You can also set up security groups in AWS so that only specifically authorized people can access the ports. That's a bit involved... dean On Fri, Jan 18, 2013 at 10:31 AM, Ariel Marcus wro= te: > Hey Tony, > > Port forwarding is the same as what Dean referred to as ssh tunneling. > > Here is a website explaining how to set up port forwarding in putty (use > port 9999): > http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html > > You can edit either hive-default or hive-site but the general practice is > to store changes from the default configuration in hive-site. > > Best, > Ariel > > > --------------------------------- > Ariel Marcus, Consultant > www.openbi.com | ariel.marcus@openbi.com > 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601 > Cell: 314-827-4356 > > > On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton = wrote: > >> Hi Ariel,**** >> >> ** ** >> >> Thanks for the speedy reply. We=92ll be accessing the HWI from Windows >> rather Linux desktops, so can you help me out with understanding how to >> carry out the port forwarding from a Windows environment? Currently I us= e >> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY >> session.**** >> >> ** ** >> >> The changes to the hive config look identical to what I=92ve changed in = my >> hive-default.xml. Does it make any difference which config file the >> changesare applied to (hive-default.xml vs hive-site.xml)?**** >> >> ** ** >> >> Tony**** >> >> ** ** >> >> ** ** >> >> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com] >> *Sent:* 18 January 2013 16:16 >> >> *To:* user@hive.apache.org >> *Subject:* Re: HWI use on AWS/EMR**** >> >> ** ** >> >> Hey Tony,**** >> >> ** ** >> >> I would recommend using ssh port forwarding instead of making your hwi >> publicly available on the internet. When you connect to the master node >> using ssh you should use a command like the following:**** >> >> ** ** >> >> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop >> MASTER_HOSTNAME**** >> >> ** ** >> >> After you have connected, add the following to your >> hive/conf/hive-site.xml file:**** >> >> ** ** >> >> **** >> >> hive.hwi.war.file**** >> >> lib/hive-hwi-0.8.1.war**** >> >> This is the WAR file with the jsp content for Hive Web >> Interface**** >> >> **** >> >> ** ** >> >> Run this command to start up hwi:**** >> >> ** ** >> >> hive --service hwi**** >> >> ** ** >> >> And finally point your browser to:**** >> >> ** ** >> >> localhost:9999**** >> >> ** ** >> >> That worked for me.**** >> >> ** ** >> >> Best,**** >> >> Ariel**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> >> **** >> >> ---------------------------------**** >> >> Ariel Marcus, Consultant**** >> >> www.openbi.com | ariel.marcus@openbi.com**** >> >> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601 >> Cell: 314-827-4356**** >> >> ** ** >> >> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler < >> dean.wampler@thinkbiganalytics.com> wrote:**** >> >> Oops, I overlooked that you have the public domain name in your message. >> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.co= m? If not, does HWI use port 80? Is whatever port it uses blocked by EC2? >> **** >> >> ** ** >> >> If it's blocked you can use ssh to tunnel the port through.**** >> >> ** ** >> >> dean**** >> >> ** ** >> >> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler < >> dean.wampler@thinkbiganalytics.com> wrote:**** >> >> That's the internal hostname, not visible outside. Use the name like >> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console >> and the elastic-mapreduce script you might have used to launch the clust= er. >> **** >> >> ** ** >> >> If that doesn't work, verify that port 80 is not blocked by default. >> That's certainly true for ports 9XXX used by the JobTracker, etc. **** >> >> ** ** >> >> dean**** >> >> ** ** >> >> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton >> wrote:**** >> >> Hi, >> >> I'm trying to get HWI running and accessible from an Amazon Web Services >> EMR instance. I've hit a blocker early on though, and the documentation = is >> less than illuminating. Can you share any experiences you have had? >> Specifically, here's what I'm curious about. >> >> - Running on AWS. I've created a Hive job flow on AWS, edited >> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed >> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war. >> - HWI starts up fine, but when I try to connect with >> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a >> timeout message, in Firefox it's "The connection has timed out - The ser= ver >> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long >> to respond". >> >> Has anyone successfully connected to HWI running on an AWS EMR instance? >> From the same browser I can connect to HWI on a local Ubuntu box. >> >> Thanks! >> >> Tony >> >> >> ********************************************************************** >> Please consider the environment before printing this email or attachment= s >> >> This email and any attachments are confidential, protected by copyright >> and may be legally privileged. If you are not the intended recipient, t= hen >> the dissemination or copying of this email is prohibited. If you have >> received this in error, please notify the sender by replying by email an= d >> then delete the email completely from your system. Neither Sporting Ind= ex >> nor the sender accepts responsibility for any virus, or any other defect >> which might affect any computer or IT system into which the email is >> received and/or opened. It is the responsibility of the recipient to sc= an >> the email and no responsibility is accepted for any loss or damage arisi= ng >> in any way from receipt or use of this email. Sporting Index Ltd is a >> company registered in England and Wales with company number 2636842, who= se >> registered office is at Gateway House, Milverton Street, London, SE11 4A= P. >> Sporting Index Ltd is authorised and regulated by the UK Financial >> Services Authority (reg. no. 150404) and Gambling Commission (reg. no. >> 000-027343-R-308898-001). Any financial promotion contained herein has >> been issued >> and approved by Sporting Index Ltd. >> >> Outbound email has been scanned for viruses and SPAM**** >> >> >> >> **** >> >> ** ** >> >> -- >> *Dean Wampler, Ph.D.***** >> >> thinkbiganalytics.com**** >> >> +1-312-339-1330**** >> >> ** ** >> >> >> >> **** >> >> ** ** >> >> -- >> *Dean Wampler, Ph.D.***** >> >> thinkbiganalytics.com**** >> >> +1-312-339-1330**** >> >> ** ** >> >> ** ** >> ------------------------------ >> >> This transmission is confidential and intended solely for the use of the >> recipient named above. It may contain confidential, proprietary, or lega= lly >> privileged information. If you are not the intended recipient, you are >> hereby notified that any unauthorized review, use, disclosure or >> distribution is strictly prohibited. If you have received this transmiss= ion >> in error, please contact the sender by reply e-mail and delete the origi= nal >> transmission and all copies from your system.**** >> >> ** ** >> >> ** ** >> ------------------------------ >> >> This transmission is confidential and intended solely for the use of the >> recipient named above. It may contain confidential, proprietary, or lega= lly >> privileged information. If you are not the intended recipient, you are >> hereby notified that any unauthorized review, use, disclosure or >> distribution is strictly prohibited. If you have received this transmiss= ion >> in error, please contact the sender by reply e-mail and delete the origi= nal >> transmission and all copies from your system.**** >> >> ** ** >> >> >> >> ************************************************************************= ***** >> P *Please consider the environment before printing this email* **** >> >> >> www.sportingindex.com >> >> Inbound email has been scanned for viruses & spam**** >> >> ------------------------------ >> >> This transmission is confidential and intended solely for the use of the >> recipient named above. It may contain confidential, proprietary, or lega= lly >> privileged information. If you are not the intended recipient, you are >> hereby notified that any unauthorized review, use, disclosure or >> distribution is strictly prohibited. If you have received this transmiss= ion >> in error, please contact the sender by reply e-mail and delete the origi= nal >> transmission and all copies from your system. >> > > > ------------------------------ > > This transmission is confidential and intended solely for the use of the > recipient named above. It may contain confidential, proprietary, or legal= ly > privileged information. If you are not the intended recipient, you are > hereby notified that any unauthorized review, use, disclosure or > distribution is strictly prohibited. If you have received this transmissi= on > in error, please contact the sender by reply e-mail and delete the origin= al > transmission and all copies from your system. > --=20 *Dean Wampler, Ph.D.* thinkbiganalytics.com +1-312-339-1330 --f46d041826aac46d8104d392be89 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yes, this is straightforward to set up in Putty. =A0Ariel, thanks for provi= ding the details I was too lazy to mention ;)

One flaw w= ith this approach is that only your machine will have access to HWI. That i= s, the port is forwarded only to your machine. Making the ports public on t= he master node means anyone on the interwebs can see it. You can also set u= p security groups in AWS so that only specifically authorized people can ac= cess the ports. That's a bit involved...

dean

On Fri, Jan 18, = 2013 at 10:31 AM, Ariel Marcus <ariel.marcus@openbi.com> wrote:
Hey Tony,

Port forwarding is the same as what Dean referred to as ssh tunneling.

Here is a website explaining how to set up port forward= ing in putty (use port 9999):

You can edit either hive-default = or hive-site but the general practice is to store changes from the default = configuration in hive-site.

Best,
Ariel


-------------= --------------------
Ariel Marcus, Consultant
150 N=A0Michigan=A0Avenue, Suite 2800, Chicago, IL 60601
Cell:=A0314-827-4356


On Fri, Jan= 18, 2013 at 11:26 AM, Tony Burton <TBurton@sportingindex.com&= gt; wrote:

Hi Ariel,

=A0

Thanks for the speedy reply. We=92ll be = accessing the HWI from Windows rather Linux desktops, so can you help me ou= t with understanding how to carry out the port forwarding from a Windows en= vironment? Currently I use WinSCP to initiate a connection to the EMR insta= nce, then spawn a PuTTY session.

=A0

The changes to the hive config look identical to wha= t I=92ve changed in my hive-default.xml. Does it make any difference which = config file the changesare applied to (hive-default.xml vs hive-site.xml)?<= u>

=A0

Tony

=A0

=A0

From: Ariel Marcus [mailto:ariel.marcus@openbi.com]
Sent: 18 January 2013 16:16


To: user@hive.apache.orgSubject: Re: HWI use on AWS/EMR

=A0

Hey= Tony,

=A0=

I would recommend using ssh port = forwarding instead of making your hwi publicly available on the internet. W= hen you connect to the master node using ssh you should use a command like = the following:

=A0

ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop= MASTER_HOSTNAME

= =A0

After you have connected, add the followi= ng to your hive/conf/hive-site.xml file:

=A0

<property>

=A0 <name>hive.hwi.war.file</nam= e>

=A0 <value>l= ib/hive-hwi-0.8.1.war</value>

=A0 <description>This is the WAR file with the jsp content for Hive W= eb Interface</description>

</property>

=A0

Run this command to = start up hwi:

=A0=

hive --service hwi

=A0

And finally point your browser to:

=A0

localhost:9999

= =A0

That worked for me.<= u>

=A0

Best,

Ariel

=A0

=A0

=A0

=A0

=A0


-----------------------------= ----

Ariel Marcus, Consultant

1= 50 N=A0Michigan=A0Avenue, Suite 2800, Chicago, IL 60601
Cell:=A0314-827-4356

=A0

On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <dean.wampler@think= biganalytics.com> wrote:

Oops, I overlooked that you have the public domain n= ame in your message. Can you surf to =A0http://ec2-54-247-61-20= 6.eu-west-1.compute.amazonaws.com ? =A0If not, does HWI use port 80? Is= whatever port it uses blocked by EC2?

=A0

If it's blocked you can use ssh to tunnel the port through.<= /u>

=A0

dean<= /span>

=A0

On Fri= , Jan 18, 2013 at 10:06 AM, Dean Wampler <dean.wampler@thinkbiganalytics.co= m> wrote:

If that doesn't work, verify that port 80 is not blocked by de= fault. That's certainly true for ports 9XXX used by the JobTracker, etc= .=A0

=A0

dean

=A0

On Fri,= Jan 18, 2013 at 9:54 AM, Tony Burton <TBurton@sportingindex.com> wrote:

Hi,

I'm trying to get HWI running and acc= essible from an Amazon Web Services EMR instance. I've hit a blocker ea= rly on though, and the documentation is less than illuminating. Can you sha= re any experiences you have had? Specifically, here's what I'm curi= ous about.

- Running on AWS. I've created a Hive job flow on AWS, edited hive-= default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hiv= e.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when= I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.c= ompute.internal:9999/hwi, I get a timeout message, in Firefox it's = "The connection has timed out - The server at ec2-54-247-6= 1-206.eu-west-1.compute.amazonaws.com is taking too long to respond&quo= t;.

Has anyone successfully connected to HWI running on an AWS EMR instance= ? From the same browser I can connect to HWI on a local Ubuntu box.

= Thanks!

Tony


********************************************= **************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyrig= ht and may be legally privileged. =A0If you are not the intended recipient,= then the dissemination or copying of this email is prohibited. If you have= received this in error, please notify the sender by replying by email and = then delete the email completely from your system. =A0Neither Sporting Inde= x nor the sender accepts responsibility for any virus, or any other defect = which might affect any computer or IT system into which the email is receiv= ed and/or opened. =A0It is the responsibility of the recipient to scan the = email and no responsibility is accepted for any loss or damage arising in a= ny way from receipt or use of this email. =A0Sporting Index Ltd is a compan= y registered in England and Wales with company number 2636842, whose regist= ered office is at Gateway House, Milverton Street, London, SE11 4AP. =A0Spo= rting Index Ltd is authorised and regulated by the UK Financial Services Au= thority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-30= 8898-001). =A0Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.

Outbound email has been scanned = for viruses and SPAM



=A0

-- <= br>Dean Wampler, Ph.D.

<= u>=A0



=A0

--
Dean Wampler, Ph.D.

=A0

=A0

This transmission is confidential and intended solely fo= r the use of the recipient named above. It may contain confidential, propri= etary, or legally privileged information. If you are not the intended recip= ient, you are hereby notified that any unauthorized review, use, disclosure= or distribution is strictly prohibited. If you have received this transmis= sion in error, please contact the sender by reply e-mail and delete the ori= ginal transmission and all copies from your system.

=A0

=A0

This transmission is confidential and intended solel= y for the use of the recipient named above. It may contain confidential, pr= oprietary, or legally privileged information. If you are not the intended r= ecipient, you are hereby notified that any unauthorized review, use, disclo= sure or distribution is strictly prohibited. If you have received this tran= smission in error, please contact the sender by reply e-mail and delete the= original transmission and all copies from your system.

=A0

=


*****************************************************= ************************
P P= lease consider the environment before printing this email


www.sportingindex.com

Inbound email has = been scanned for viruses & spam


This transmission is confidential and inten= ded solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are here= by notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, pleas= e contact the sender by reply e-mail and delete the original transmission and= all copies from your system.



This transmission is confidential and inten= ded solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are here= by notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, pleas= e contact the sender by reply e-mail and delete the original transmission and= all copies from your system.




--
Dean Wampler, Ph.D.
+1-312-339-1330

--f46d041826aac46d8104d392be89--