hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yinghua hu <yinghua...@gmail.com>
Subject Re: shutting down instance
Date Thu, 15 Nov 2012 15:29:59 GMT
Since I only have 4 nodes, I just copy and paste the external address from
amazon aws console web-interface. It is really much more work if you have
20 or more nodes in the cluster.

Other than that, I have a script to scp the xml files to all the hosts
listed in the slaves file. More nodes won't bring more work on transferring
since it is automatic.

Did you try to explore the option of VPC in AWS? I do not have experience
on that.

On Thu, Nov 15, 2012 at 6:53 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca>wrote:

>  I also use scp and remote ssh to push configuration files |
> start-stop-restart respective daemons in the cluster. This can simply
> resolution of every node in the cluster having an updated address of the
> new NN. But what about slave file on the NN?
>
>
>
> If you have 20 DN/TTs in a cluster, you will have to go inside each and
> every one to get internal ip-address or copy external ip-addresses from EC2
> web-inteface for each node and then replace the list of new nodes in your
> slave file.  Is this how you do it?
>
>
>
> Rgds,
>
> AK-47
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Wednesday, November 14, 2012 12:03 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: shutting down instance
>
>
>
> Hi,Andy
>
>
>
> Yes. I shutdown EC2 instance every day to save some money for my company.
> I use shell scripts to reduce manual editing as much as possible. I just
> edit core-site.xml and yarn-site.xml in single node and then scp it to all
> the other nodes. This can be automated. You can add to /etc/ssh/ssh-config
> the two lines below
>
>
>
> StrictHostKeyChecking no
>
> UserKnownHostsFile /dev/null
>
>
>
> so that you will not need to type "yes" when connecting to an unknown
> host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and
> master on namenode everyday. It take only a few minutes.
>
>
>
> I did not use Zookeeper to deal with this. Please let me know if you have
> better approach.
>
>
>
> On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca>
> wrote:
>
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone  successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <stevel@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yinghua.hu@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
>
>
>
>
> --
> Regards,
>
> Yinghua
>  NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>



-- 
Regards,

Yinghua

Mime
View raw message