hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From max scalf <oracle.bl...@gmail.com>
Subject Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
Date Fri, 06 Mar 2015 02:54:52 GMT
unfortunately without DNS you have to rely on /etc/hosts, so put in entry
for all your nodes(nn,snn,dn1,dn2 etc..) on all nodes(/etc/hosts file) and
i have that tested for hortonworks(using ambari) and cloudera manager and i
am certainly sure it will work for MapR

On Thu, Mar 5, 2015 at 8:47 PM, Alexander Pivovarov <apivovarov@gmail.com>
wrote:

> what about DNS?
> if you have 2 computers (nn and dn) how nn knows dn ip?
>
> The script puts only this computer ip to /etc/hosts
>
> On Thu, Mar 5, 2015 at 6:39 PM, max scalf <oracle.blog3@gmail.com> wrote:
>
>> Here is a easy way to go about assigning static name to your ec2
>> instance.  When you get the launch an EC2-instance from aws console when
>> you get to the point of selecting VPC, ip address screen there is a screen
>> that says "USER DATA"...put the below in with appropriate host name(change
>> CHANGE_HOST_NAME_HERE to whatever you want) and that should be able to get
>> you static name.
>>
>> #!/bin/bash
>>
>> HOSTNAME_TAG=CHANGE_HOST_NAME_HERE
>> cat > /etc/sysconfig/network << EOF
>> NETWORKING=yes
>> NETWORKING_IPV6=no
>> HOSTNAME=${HOSTNAME_TAG}
>> EOF
>>
>> IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
>> echo "${IP} ${HOSTNAME_TAG}.localhost ${HOSTNAME_TAG}" >> /etc/hosts
>>
>> echo ${HOSTNAME_TAG} > /proc/sys/kernel/hostname
>> service network restart
>>
>>
>> Also note i was able to do this on couple of spot instance for cheap
>> price, only thing is once you shut it down or someone outbids you, you
>> loose that instance but its easy/cheap to play around with.... and i have
>> used couple of m3.medium for my NN/SNN and couple of them for data nodes...
>>
>> On Thu, Mar 5, 2015 at 7:19 PM, Jonathan Aquilina <
>> jaquilina@eagleeyet.net> wrote:
>>
>>>  I dont know how you would do that to be honest. With EMR you have
>>> destinctions master core and task nodes. If you need to change
>>> configuration you just ssh into the EMR master node.
>>>
>>>
>>>
>>> ---
>>> Regards,
>>> Jonathan Aquilina
>>> Founder Eagle Eye T
>>>
>>>  On 2015-03-06 02:11, Alexander Pivovarov wrote:
>>>
>>> What is the easiest way to assign names to aws ec2 computers?
>>> I guess computer need static hostname and dns name before it can be used
>>> in hadoop cluster.
>>> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <jaquilina@eagleeyet.net>
>>> wrote:
>>>
>>>>  When I started with EMR it was alot of testing and trial and error.
>>>> HUE is already supported as something that can be installed from the AWS
>>>> console. What I need to know is if you need this cluster on all the time
or
>>>> this is goign ot be what amazon call a transient cluster. Meaning you fire
>>>> it up run the job and tear it back down.
>>>>
>>>>
>>>>
>>>> ---
>>>> Regards,
>>>> Jonathan Aquilina
>>>> Founder Eagle Eye T
>>>>
>>>>  On 2015-03-06 01:10, Krish Donald wrote:
>>>>
>>>>  Thanks Jonathan,
>>>>
>>>> I will try to explore EMR option also.
>>>> Can you please let me know the configuration which you have used it?
>>>> Can you please recommend for me also?
>>>> I would like to setup Hadoop cluster using cloudera manager and then
>>>> would like to do below things:
>>>>
>>>> setup kerberos
>>>> setup federation
>>>> setup monitoring
>>>> setup hadr
>>>> backup and recovery
>>>> authorization using sentry
>>>> backup and recovery of individual componenets
>>>> performamce tuning
>>>> upgrade of cdh
>>>> upgrade of CM
>>>> Hue User Administration
>>>> Spark
>>>> Solr
>>>>
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>>
>>>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina <
>>>> jaquilina@eagleeyet.net> wrote:
>>>>
>>>>>  krish EMR wont cost you much with all the testing and data we ran
>>>>> through the test systems as well as the large amont of data when everythign
>>>>> was read we paid about 15.00 USD. I honestly do not think that the specs
>>>>> there would be enough as java can be pretty ram hungry.
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Regards,
>>>>> Jonathan Aquilina
>>>>> Founder Eagle Eye T
>>>>>
>>>>>   On 2015-03-06 00:41, Krish Donald wrote:
>>>>>
>>>>>  Hi,
>>>>>
>>>>> I am new to AWS and would like to setup Hadoop cluster using cloudera
>>>>> manager for 6-7 nodes.
>>>>>
>>>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
>>>>> I would like to use free service as of now.
>>>>>
>>>>> Please advise.
>>>>>
>>>>> Thanks
>>>>> Krish
>>>>>
>>>>>
>>
>

Mime
View raw message