Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E389B17C70 for ; Fri, 6 Mar 2015 02:56:52 +0000 (UTC) Received: (qmail 82378 invoked by uid 500); 6 Mar 2015 02:56:47 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 82251 invoked by uid 500); 6 Mar 2015 02:56:47 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 82205 invoked by uid 99); 6 Mar 2015 02:56:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2015 02:56:47 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of oracle.blog3@gmail.com designates 209.85.213.173 as permitted sender) Received: from [209.85.213.173] (HELO mail-ig0-f173.google.com) (209.85.213.173) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2015 02:56:42 +0000 Received: by igdh15 with SMTP id h15so623118igd.4 for ; Thu, 05 Mar 2015 18:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Zs91ejVY66u43G3poxKerlQcWyD0R3VhhygQ8ny+KYA=; b=p76OBmx/m+RJ4qx//dGGHRX6XOddCmt7GPApnkBfajEk3dOjpzubp7zMzSlzQBGct9 zRzTol5GT3Qvd1I1qD/QQ6lCiwiyfwEbK2WZcQLwsrA3tbPTJlj5oF7X1Qi5bzFU8nEZ 740zNz79fd4FKZ54RJBnxpOo6jpO0WUCRa5vosV+Ybs6S7U294wAjynGbBKrIRqXFRKl MFLTU3IfP18F3di1+qKwT1LlMtMvTAq6u7BgjXxPo7qZ4vlLKgBvArn+LHkPyfVJi2w2 0pOlXuvtu+ijNjrZbtSMvsJKoYxD4/GietYwml+Sty4mZK9AAzfuWVLKqEq7+ybb3Ua8 QmBw== MIME-Version: 1.0 X-Received: by 10.107.16.42 with SMTP id y42mr26024446ioi.41.1425610492265; Thu, 05 Mar 2015 18:54:52 -0800 (PST) Received: by 10.107.17.145 with HTTP; Thu, 5 Mar 2015 18:54:52 -0800 (PST) In-Reply-To: References: <0a49e536b65de41e39692a05333e8254@mail.eagleeyet.net> <285b89b414575517a0181dae0f6dfd87@mail.eagleeyet.net> <9cdd28c6b02494f8ae91cd9abc50d276@mail.eagleeyet.net> Date: Thu, 5 Mar 2015 20:54:52 -0600 Message-ID: Subject: Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ? From: max scalf To: HDP mailing list Content-Type: multipart/alternative; boundary=001a113f1b24facc20051095ce62 X-Virus-Checked: Checked by ClamAV on apache.org --001a113f1b24facc20051095ce62 Content-Type: text/plain; charset=UTF-8 unfortunately without DNS you have to rely on /etc/hosts, so put in entry for all your nodes(nn,snn,dn1,dn2 etc..) on all nodes(/etc/hosts file) and i have that tested for hortonworks(using ambari) and cloudera manager and i am certainly sure it will work for MapR On Thu, Mar 5, 2015 at 8:47 PM, Alexander Pivovarov wrote: > what about DNS? > if you have 2 computers (nn and dn) how nn knows dn ip? > > The script puts only this computer ip to /etc/hosts > > On Thu, Mar 5, 2015 at 6:39 PM, max scalf wrote: > >> Here is a easy way to go about assigning static name to your ec2 >> instance. When you get the launch an EC2-instance from aws console when >> you get to the point of selecting VPC, ip address screen there is a screen >> that says "USER DATA"...put the below in with appropriate host name(change >> CHANGE_HOST_NAME_HERE to whatever you want) and that should be able to get >> you static name. >> >> #!/bin/bash >> >> HOSTNAME_TAG=CHANGE_HOST_NAME_HERE >> cat > /etc/sysconfig/network << EOF >> NETWORKING=yes >> NETWORKING_IPV6=no >> HOSTNAME=${HOSTNAME_TAG} >> EOF >> >> IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4) >> echo "${IP} ${HOSTNAME_TAG}.localhost ${HOSTNAME_TAG}" >> /etc/hosts >> >> echo ${HOSTNAME_TAG} > /proc/sys/kernel/hostname >> service network restart >> >> >> Also note i was able to do this on couple of spot instance for cheap >> price, only thing is once you shut it down or someone outbids you, you >> loose that instance but its easy/cheap to play around with.... and i have >> used couple of m3.medium for my NN/SNN and couple of them for data nodes... >> >> On Thu, Mar 5, 2015 at 7:19 PM, Jonathan Aquilina < >> jaquilina@eagleeyet.net> wrote: >> >>> I dont know how you would do that to be honest. With EMR you have >>> destinctions master core and task nodes. If you need to change >>> configuration you just ssh into the EMR master node. >>> >>> >>> >>> --- >>> Regards, >>> Jonathan Aquilina >>> Founder Eagle Eye T >>> >>> On 2015-03-06 02:11, Alexander Pivovarov wrote: >>> >>> What is the easiest way to assign names to aws ec2 computers? >>> I guess computer need static hostname and dns name before it can be used >>> in hadoop cluster. >>> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" >>> wrote: >>> >>>> When I started with EMR it was alot of testing and trial and error. >>>> HUE is already supported as something that can be installed from the AWS >>>> console. What I need to know is if you need this cluster on all the time or >>>> this is goign ot be what amazon call a transient cluster. Meaning you fire >>>> it up run the job and tear it back down. >>>> >>>> >>>> >>>> --- >>>> Regards, >>>> Jonathan Aquilina >>>> Founder Eagle Eye T >>>> >>>> On 2015-03-06 01:10, Krish Donald wrote: >>>> >>>> Thanks Jonathan, >>>> >>>> I will try to explore EMR option also. >>>> Can you please let me know the configuration which you have used it? >>>> Can you please recommend for me also? >>>> I would like to setup Hadoop cluster using cloudera manager and then >>>> would like to do below things: >>>> >>>> setup kerberos >>>> setup federation >>>> setup monitoring >>>> setup hadr >>>> backup and recovery >>>> authorization using sentry >>>> backup and recovery of individual componenets >>>> performamce tuning >>>> upgrade of cdh >>>> upgrade of CM >>>> Hue User Administration >>>> Spark >>>> Solr >>>> >>>> >>>> Thanks >>>> Krish >>>> >>>> >>>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina < >>>> jaquilina@eagleeyet.net> wrote: >>>> >>>>> krish EMR wont cost you much with all the testing and data we ran >>>>> through the test systems as well as the large amont of data when everythign >>>>> was read we paid about 15.00 USD. I honestly do not think that the specs >>>>> there would be enough as java can be pretty ram hungry. >>>>> >>>>> >>>>> >>>>> --- >>>>> Regards, >>>>> Jonathan Aquilina >>>>> Founder Eagle Eye T >>>>> >>>>> On 2015-03-06 00:41, Krish Donald wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am new to AWS and would like to setup Hadoop cluster using cloudera >>>>> manager for 6-7 nodes. >>>>> >>>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ? >>>>> I would like to use free service as of now. >>>>> >>>>> Please advise. >>>>> >>>>> Thanks >>>>> Krish >>>>> >>>>> >> > --001a113f1b24facc20051095ce62 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
unfortunately without DNS you have to rely= on /etc/hosts, so put in entry for all your nodes(nn,snn,dn1,dn2 etc..) on= all nodes(/etc/hosts file) and i have that tested for hortonworks(using am= bari) and cloudera manager and i am certainly sure it will work for MapR

On Thu, = Mar 5, 2015 at 8:47 PM, Alexander Pivovarov <apivovarov@gmail.com= > wrote:
= what about DNS?
if you have 2 computers (nn and dn) how nn knows dn ip?<= br>

The script puts only this computer ip to /etc/hosts
=

On Thu, Mar 5, 2015 at 6:39 PM, max scalf <or= acle.blog3@gmail.com> wrote:
Here is a easy way to go about= assigning static name to your ec2 instance.=C2=A0 When you get the launch = an EC2-instance from aws console when you get to the point of selecting VPC= , ip address screen there is a screen that says "USER DATA"...put= the below in with appropriate host name(change CHANGE_HOST_NAME_HERE to wh= atever you want) and that should be able to get you static name. =C2=A0

#!/bin/bash=

HOSTNAME_TAG= =3DCHANGE_HOST_NAME_HERE
cat > /etc/sysconfig/network <= ;< EOF
NETWORKING=3Dyes
NETWORKING_IPV6=3D= no
HOSTNAME=3D${HOSTNAME_TAG}
EOF

IP=3D$(curl http://169.254.169.254/latest/met= a-data/local-ipv4)
echo "${IP} ${HOSTNAME_TAG}.localhost ${HOSTNAME_TAG}" &g= t;> /etc/hosts

echo ${HOSTNAME_TAG} &= gt; /proc/sys/kernel/hostname
service network restart=


= Also note i was able to do= this on couple of spot instance for cheap price, only thing is once you sh= ut it down or someone outbids you, you loose that instance but its easy/che= ap to play around with.... and i have used couple of m3.medium for my NN/SN= N and couple of them for data nodes...

On Thu, Mar 5, 2015= at 7:19 PM, Jonathan Aquilina <jaquilina@eagleeyet.net> wrote:

I dont know how you would do that to be honest. With EMR you have destin= ctions master core and task nodes. If you need to change configuration you = just ssh into the EMR master node.

=C2=A0

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-03-06 02:11, Alexander Pivovarov wrote:

What is the easiest way to assign names to aws ec2 computers= ?
I guess computer need static hostname and dns name before it can be u= sed in hadoop cluster.

On Mar 5, 2015 4:36 PM, "Jonathan Aquilina&= quot; <jaqu= ilina@eagleeyet.net> wrote:

When I started with EMR it was alot of testing and trial and error. HUE = is already supported as something that can be installed from the AWS consol= e. What I need to know is if you need this cluster on all the time or this = is goign ot be what amazon call a transient cluster. Meaning you fire it up= run the job and tear it back down.

=C2=A0

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-03-06 01:10, Krish Donald wrote:

Thanks Jonathan,
=C2=A0
I will try to explore EMR option also.
Can you please let me know the configuration which you have used it?
Can you please recommend for me also?
I would like to setup Hadoop cluster using cloudera manager and then w= ould like to do below things:
=C2=A0
setup kerberos
setup federation
setup monitoring
setup hadrbackup and recovery
authorization using sentry
backup and recovery = of individual componenets
performamce tuning
upgrade of cdh
upgra= de of CM
Hue User Administration
Spark
Solr
=C2=A0
=C2=A0
Thanks
Krish
=C2=A0

On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilin= a <ja= quilina@eagleeyet.net> wrote:

krish EMR wont cost you much with all the testing and data we ran throug= h the test systems as well as the large amont of data when everythign was r= ead we paid about 15.00 USD. I honestly do not think that the specs there w= ould be enough as java can be pretty ram hungry.

=C2=A0

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-03-06 00:41, Krish Donald wrote:

Hi,
=C2=A0
I am new to AWS and would like to setup Hadoop cluster using cloudera = manager for 6-7 nodes.
=C2=A0
t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
I would like to use free service as of now.
=C2=A0
Please advise.
=C2=A0
Thanks
Krish



--001a113f1b24facc20051095ce62--