hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Pivovarov <apivova...@gmail.com>
Subject Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
Date Fri, 06 Mar 2015 02:35:18 GMT
I found the following solution to this problem

I registered 2 subdomains  (public and local) for each computer on
https://freedns.afraid.org/subdomain/
e.g.
myhadoop-nn.crabdance.com
myhadoop-nn-local.crabdance.com

then I added cron job which sends http requests to update public and local
ip on freedns server
hint: public ip is detected automatically
ip address for local name can be set using request parameter
&address=10.x.x.x
(don't forget to escape &)

as a result my nn computer has 2 DNS names with currently assigned ip
addresses , e.g.
myhadoop-nn.crabdance.com  54.203.181.177
myhadoop-nn-local.crabdance.com   10.220.149.103

in hadoop configuration I can use local machine names
to access my cluster outside of AWS I can use public names

Just curious if AWS provides easier way to name EC2 computers?

On Thu, Mar 5, 2015 at 5:19 PM, Jonathan Aquilina <jaquilina@eagleeyet.net>
wrote:

>  I dont know how you would do that to be honest. With EMR you have
> destinctions master core and task nodes. If you need to change
> configuration you just ssh into the EMR master node.
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
>  On 2015-03-06 02:11, Alexander Pivovarov wrote:
>
> What is the easiest way to assign names to aws ec2 computers?
> I guess computer need static hostname and dns name before it can be used
> in hadoop cluster.
> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <jaquilina@eagleeyet.net>
> wrote:
>
>>  When I started with EMR it was alot of testing and trial and error. HUE
>> is already supported as something that can be installed from the AWS
>> console. What I need to know is if you need this cluster on all the time or
>> this is goign ot be what amazon call a transient cluster. Meaning you fire
>> it up run the job and tear it back down.
>>
>>
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>>  On 2015-03-06 01:10, Krish Donald wrote:
>>
>>  Thanks Jonathan,
>>
>> I will try to explore EMR option also.
>> Can you please let me know the configuration which you have used it?
>> Can you please recommend for me also?
>> I would like to setup Hadoop cluster using cloudera manager and then
>> would like to do below things:
>>
>> setup kerberos
>> setup federation
>> setup monitoring
>> setup hadr
>> backup and recovery
>> authorization using sentry
>> backup and recovery of individual componenets
>> performamce tuning
>> upgrade of cdh
>> upgrade of CM
>> Hue User Administration
>> Spark
>> Solr
>>
>>
>> Thanks
>> Krish
>>
>>
>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina <
>> jaquilina@eagleeyet.net> wrote:
>>
>>>  krish EMR wont cost you much with all the testing and data we ran
>>> through the test systems as well as the large amont of data when everythign
>>> was read we paid about 15.00 USD. I honestly do not think that the specs
>>> there would be enough as java can be pretty ram hungry.
>>>
>>>
>>>
>>> ---
>>> Regards,
>>> Jonathan Aquilina
>>> Founder Eagle Eye T
>>>
>>>   On 2015-03-06 00:41, Krish Donald wrote:
>>>
>>>  Hi,
>>>
>>> I am new to AWS and would like to setup Hadoop cluster using cloudera
>>> manager for 6-7 nodes.
>>>
>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
>>> I would like to use free service as of now.
>>>
>>> Please advise.
>>>
>>> Thanks
>>> Krish
>>>
>>>

Mime
View raw message