hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: HWI use on AWS/EMR
Date Mon, 21 Jan 2013 13:16:54 GMT
if you are not inside VPC ec2 does not let you change security groups.

so best option would be create a new ami from ur instance and then launch a
new instance with a security group you newly created.
or change the default security group to have the changes you want




On Mon, Jan 21, 2013 at 6:43 PM, Tony Burton <TBurton@sportingindex.com>wrote:

> Thanks for the suggestion Nitin.****
>
> ** **
>
> I can see the “Change Security Groups” option for my instance (when
> looking at the EC2 console), but it’s greyed out. The only option I have on
> the EMR console for the instance is “Terminate Job”.****
>
> ** **
>
> Is your “Change Security Groups” option enabled? If so, any idea what
> steps you took in creating the instance to enable it?****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 21 January 2013 11:57
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ** **
>
> right click on the instance in your aws console and you  should be able to
> change the security group for the node. ****
>
> ** **
>
> This feature was not there previously, came in few months back ****
>
> ** **
>
> On Mon, Jan 21, 2013 at 5:09 PM, Tony Burton <TBurton@sportingindex.com>
> wrote:****
>
> I’ve tracked down this document on AWS:****
>
>  ****
>
>
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html
> ****
>
>  ****
>
> which shows how to create security groups to associate with your AWS
> instances. Simple enough to create a rule – but is there anyway to
> associate a rule with an instance either when the instance is running, or
> at creation time using the AWS Management Console? Or can it only be done
> using the commandline client or by an http request? ****
>
>  ****
>
> Thanks,****
>
>  ****
>
> Tony****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 17:25****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
>  ****
>
> Hey Tony,****
>
>  ****
>
> Nitin's approach would work for you as well. Amazon has very good
> documentation of their services. A quick google search should get you the
> right info.****
>
>  ****
>
> Best,****
>
> Ariel****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
>  ****
>
> On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TBurton@sportingindex.com>
> wrote:****
>
> If you could provide the steps to do this I’d be grateful – thanks.****
>
>  ****
>
> Thanks for all your input so far, time to head home for the weekend so
> I’ll try it again with a fresh EMR instance next week.****
>
>  ****
>
> Tony****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 18 January 2013 17:08****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
>  ****
>
> ok so much discussion around this but why dont you open access to port
> 9999 for your specific machine (home/office) and access it normally without
> setting up port forwarding. ****
>
>  ****
>
> AWS does give you ip level access control in the security groups ****
>
>  ****
>
> On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ariel.marcus@openbi.com>
> wrote:****
>
> This should be localhost instead of proxy:****
>
>  ****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box, *
> localhost*:9999 in the Destination box. Not sure if this is correct.****
>
>  ****
>
> Best,****
>
> Aril****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
>  ****
>
> On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TBurton@sportingindex.com>
> wrote:****
>
>  ****
>
> Ok, thanks Ariel (and Dean in the next email!)****
>
>  ****
>
> I’ve set up PuTTY (v0.62) as follows:****
>
> - EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
> ****
>
> - In Connection/SSH/Auth I’ve added my private key file, downloaded from
> AWS ages for this account.****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box,
> proxy:9999 in the Destination box. Not sure if this is correct.****
>
>  ****
>
> Hit the Open button, I can connect. (And I can see my changes in
> hive-default.xml, so I know I’m in the right place:)****
>
> Then pointing my browser at http://localhost:9999 I get the same timeout
> again. Have I set up everything right?****
>
>  ****
>
> Also, following good advice I copied the config changes to hive-site.xml.
> How do I stop hive so I can restart with ‘hive –service hwi’ so that it
> picks up the new config?****
>
>  ****
>
> Tony****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:31****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
>  ****
>
> Hey Tony,****
>
>  ****
>
> Port forwarding is the same as what Dean referred to as ssh tunneling.****
>
>  ****
>
> Here is a website explaining how to set up port forwarding in putty (use
> port 9999):****
>
> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html****
>
>  ****
>
> You can edit either hive-default or hive-site but the general practice is
> to store changes from the default configuration in hive-site.****
>
>  ****
>
> Best,****
>
> Ariel****
>
>  ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
>  ****
>
> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TBurton@sportingindex.com>
> wrote:****
>
> Hi Ariel,****
>
>  ****
>
> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
> rather Linux desktops, so can you help me out with understanding how to
> carry out the port forwarding from a Windows environment? Currently I use
> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
> session.****
>
>  ****
>
> The changes to the hive config look identical to what I’ve changed in my
> hive-default.xml. Does it make any difference which config file the
> changesare applied to (hive-default.xml vs hive-site.xml)?****
>
>  ****
>
> Tony****
>
>  ****
>
>  ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:16****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
>  ****
>
> Hey Tony,****
>
>  ****
>
> I would recommend using ssh port forwarding instead of making your hwi
> publicly available on the internet. When you connect to the master node
> using ssh you should use a command like the following:****
>
>  ****
>
> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
> MASTER_HOSTNAME****
>
>  ****
>
> After you have connected, add the following to your
> hive/conf/hive-site.xml file:****
>
>  ****
>
> <property>****
>
>   <name>hive.hwi.war.file</name>****
>
>   <value>lib/hive-hwi-0.8.1.war</value>****
>
>   <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>****
>
> </property>****
>
>  ****
>
> Run this command to start up hwi:****
>
>  ****
>
> hive --service hwi****
>
>  ****
>
> And finally point your browser to:****
>
>  ****
>
> localhost:9999****
>
>  ****
>
> That worked for me.****
>
>  ****
>
> Best,****
>
> Ariel****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
>  ****
>
> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to  http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com?  If not, does
HWI use port 80? Is whatever port it uses blocked by EC2?
> ****
>
>  ****
>
> If it's blocked you can use ssh to tunnel the port through.****
>
>  ****
>
> dean****
>
>  ****
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
> ****
>
>  ****
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>
>  ****
>
> dean****
>
>  ****
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TBurton@sportingindex.com>
> wrote:****
>
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged.  If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system.  Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened.  It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email.  Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
>  Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001).  Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM****
>
>
>
> ****
>
>  ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
>  ****
>
>
>
> ****
>
>  ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
>  ****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>  ****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>  ****
>
>
>
> *****************************************************************************
> P *Please consider the environment before printing this email* ****
>
>
> www.sportingindex.com
>
> Inbound email has been scanned for viruses & spam****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>  ****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>  ****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>
>
> ****
>
>  ****
>
> --
> Nitin Pawar****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>  ****
>
>  ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>



-- 
Nitin Pawar

Mime
View raw message