Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5D2AD17BC9 for ; Fri, 6 Mar 2015 02:36:56 +0000 (UTC) Received: (qmail 26573 invoked by uid 500); 6 Mar 2015 02:36:50 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 26464 invoked by uid 500); 6 Mar 2015 02:36:50 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 26453 invoked by uid 99); 6 Mar 2015 02:36:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2015 02:36:49 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_TRY_3LD X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of apivovarov@gmail.com designates 209.85.215.54 as permitted sender) Received: from [209.85.215.54] (HELO mail-la0-f54.google.com) (209.85.215.54) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2015 02:36:25 +0000 Received: by labgm9 with SMTP id gm9so15304714lab.7 for ; Thu, 05 Mar 2015 18:35:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=1A6Qiv0+AJX0/7RwJCa5HUxdb5nuncxBZo4pt8aJ+cM=; b=Hkg1/8RhI1Kht9NtBcl2h9dO4FTE7v2401mIeKhK6y6O/RCNIl+cptwNAZhkwVViqb F8yehfdy5yAGjnk3DVA/Npm+MeBp/8XPQSwSKinOkZ/wrgTd5yPhJoPtDdDTZn/xiL0u w6dbRReXGg7ndRje7gCfDz0PsYEYXvgOYUGuOSJFYZRwu5aW/i76EivTwyhzpmfzIiK6 SvNncgHHSHRQ3pelRtJt73gwbV7Zkyj0hV/UFyPjGTofH2Cqcx+1ZR0DW3S/9JwyEoKE z7vU32i6fsB2PaAvcXu+nBIkgSDxDl9/gCzum5UH2G0qU3XgLty8S/nvqH+VoWQnBO36 kgZA== X-Received: by 10.152.43.67 with SMTP id u3mr10961027lal.10.1425609338820; Thu, 05 Mar 2015 18:35:38 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.151.16 with HTTP; Thu, 5 Mar 2015 18:35:18 -0800 (PST) In-Reply-To: <9cdd28c6b02494f8ae91cd9abc50d276@mail.eagleeyet.net> References: <0a49e536b65de41e39692a05333e8254@mail.eagleeyet.net> <285b89b414575517a0181dae0f6dfd87@mail.eagleeyet.net> <9cdd28c6b02494f8ae91cd9abc50d276@mail.eagleeyet.net> From: Alexander Pivovarov Date: Thu, 5 Mar 2015 18:35:18 -0800 Message-ID: Subject: Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ? To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c242de3aa0690510958a11 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c242de3aa0690510958a11 Content-Type: text/plain; charset=UTF-8 I found the following solution to this problem I registered 2 subdomains (public and local) for each computer on https://freedns.afraid.org/subdomain/ e.g. myhadoop-nn.crabdance.com myhadoop-nn-local.crabdance.com then I added cron job which sends http requests to update public and local ip on freedns server hint: public ip is detected automatically ip address for local name can be set using request parameter &address=10.x.x.x (don't forget to escape &) as a result my nn computer has 2 DNS names with currently assigned ip addresses , e.g. myhadoop-nn.crabdance.com 54.203.181.177 myhadoop-nn-local.crabdance.com 10.220.149.103 in hadoop configuration I can use local machine names to access my cluster outside of AWS I can use public names Just curious if AWS provides easier way to name EC2 computers? On Thu, Mar 5, 2015 at 5:19 PM, Jonathan Aquilina wrote: > I dont know how you would do that to be honest. With EMR you have > destinctions master core and task nodes. If you need to change > configuration you just ssh into the EMR master node. > > > > --- > Regards, > Jonathan Aquilina > Founder Eagle Eye T > > On 2015-03-06 02:11, Alexander Pivovarov wrote: > > What is the easiest way to assign names to aws ec2 computers? > I guess computer need static hostname and dns name before it can be used > in hadoop cluster. > On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" > wrote: > >> When I started with EMR it was alot of testing and trial and error. HUE >> is already supported as something that can be installed from the AWS >> console. What I need to know is if you need this cluster on all the time or >> this is goign ot be what amazon call a transient cluster. Meaning you fire >> it up run the job and tear it back down. >> >> >> >> --- >> Regards, >> Jonathan Aquilina >> Founder Eagle Eye T >> >> On 2015-03-06 01:10, Krish Donald wrote: >> >> Thanks Jonathan, >> >> I will try to explore EMR option also. >> Can you please let me know the configuration which you have used it? >> Can you please recommend for me also? >> I would like to setup Hadoop cluster using cloudera manager and then >> would like to do below things: >> >> setup kerberos >> setup federation >> setup monitoring >> setup hadr >> backup and recovery >> authorization using sentry >> backup and recovery of individual componenets >> performamce tuning >> upgrade of cdh >> upgrade of CM >> Hue User Administration >> Spark >> Solr >> >> >> Thanks >> Krish >> >> >> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina < >> jaquilina@eagleeyet.net> wrote: >> >>> krish EMR wont cost you much with all the testing and data we ran >>> through the test systems as well as the large amont of data when everythign >>> was read we paid about 15.00 USD. I honestly do not think that the specs >>> there would be enough as java can be pretty ram hungry. >>> >>> >>> >>> --- >>> Regards, >>> Jonathan Aquilina >>> Founder Eagle Eye T >>> >>> On 2015-03-06 00:41, Krish Donald wrote: >>> >>> Hi, >>> >>> I am new to AWS and would like to setup Hadoop cluster using cloudera >>> manager for 6-7 nodes. >>> >>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ? >>> I would like to use free service as of now. >>> >>> Please advise. >>> >>> Thanks >>> Krish >>> >>> --001a11c242de3aa0690510958a11 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I found the following soluti= on to this problem

I registered 2 subdomains=C2=A0 (public and local= ) for each computer on https://freedns.afraid.org/subdomain/

then I added cron job which sends http reques= ts to update public and local ip on freedns server
hint: public ip= is detected automatically
ip address for local name can be set us= ing request parameter &address=3D10.x.x.x=C2=A0=C2=A0 (don't forget to es= cape &)

as a result my nn computer has 2 DNS names with c= urrently assigned ip addresses , e.g.
myhadoop-nn.crabdance.com=C2=A0 54.203.181.177
myhadoop-nn-local.crabdance= .com=C2=A0=C2=A0 10.220.149.103

in hadoop configuration I = can use local machine names
to access my cluster outside of AWS I = can use public names

Just curious if AWS provid= es easier way to name EC2 computers?

On Thu, Mar 5, 2015 a= t 5:19 PM, Jonathan Aquilina <jaquilina@eagleeyet.net>= wrote:

I dont know how you would do that to be honest. With EMR you have destin= ctions master core and task nodes. If you need to change configuration you = just ssh into the EMR master node.

=C2=A0

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-03-06 02:11, Alexander Pivovarov w= rote:

What is the easiest way to assign names to aws ec2 computers= ?
I guess computer need static hostname and dns name before it can be u= sed in hadoop cluster.

On Mar 5, 2015 4:36 PM, "Jonathan Aquilina&= quot; <jaqu= ilina@eagleeyet.net> wrote:

When I started with EMR it was alot of testing and trial and error. HUE = is already supported as something that can be installed from the AWS consol= e. What I need to know is if you need this cluster on all the time or this = is goign ot be what amazon call a transient cluster. Meaning you fire it up= run the job and tear it back down.

=C2=A0

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-03-06 01:10, Krish Donald wrote:

Thanks Jonathan,
=C2=A0
I will try to explore EMR option also.
Can you please let me know the configuration which you have used it?
Can you please recommend for me also?
I would like to setup Hadoop cluster using cloudera manager and then w= ould like to do below things:
=C2=A0
setup kerberos
setup federation
setup monitoring
setup hadrbackup and recovery
authorization using sentry
backup and recovery = of individual componenets
performamce tuning
upgrade of cdh
upgra= de of CM
Hue User Administration
Spark
Solr
=C2=A0
=C2=A0
Thanks
Krish
=C2=A0

On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilin= a <ja= quilina@eagleeyet.net> wrote:

krish EMR wont cost you much with all the testing and data we ran throug= h the test systems as well as the large amont of data when everythign was r= ead we paid about 15.00 USD. I honestly do not think that the specs there w= ould be enough as java can be pretty ram hungry.

=C2=A0

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-03-06 00:41, Krish Donald wrote:

Hi,
=C2=A0
I am new to AWS and would like to setup Hadoop cluster using cloudera = manager for 6-7 nodes.
=C2=A0
t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
I would like to use free service as of now.
=C2=A0
Please advise.
=C2=A0
Thanks
Krish

--001a11c242de3aa0690510958a11--