hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From max scalf <oracle.bl...@gmail.com>
Subject Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
Date Fri, 06 Mar 2015 14:09:18 GMT
@jonathan,

I totaly agree that this is reinventing the wheel, but think about the
folks who wants to do this setup from scratch to better under hadoop or
maybe those folks who are going to do admin realted work...and hence the
need to setting is up from scratch...

@alexandar,

Yes you are right, if one time effort of setting up freedns but for me it
was easy enough coz i gave static hostname thru the user data script and
also static ip address for each host...once that was done, the way i pushed
out /etc/hosts was below...on lets say the master node i edited /etc/hosts
file and put all my other nodes info on there...next setup SSH(as we have
to do this anyways) for hadoop install, once SSH is setup just create a new
file calle hosts.txt and put all your hostname in there and run a for loop
like below....

for host in `cat hosts.txt`; do
    scp /etc/hosts root@$host:/etc/hosts
done

when i frist was getting started on HDP i used the below link which helped
me, it pushes out /etc/hosts file and also does other stuff...check it out

http://sacharya.com/deploying-multinode-hadoop-20-cluster-using-apache-ambari/




On Fri, Mar 6, 2015 at 12:43 AM, Jonathan Aquilina <jaquilina@eagleeyet.net>
wrote:

>  The only limitation I know is that of how many nodes you can have and
> how many instances of that particular size the host is on can support. you
> can load hive in EMR and then any other features of the cluster are managed
> at the master node level as you have SSH access there.
>
> What are the advantage of 2.6 over 2.4 for example.
>
> I just feel you guys are reinventing the wheel when amazon already caters
> for hadoop granted it might not be 2.6.
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
>  On 2015-03-06 07:31, Alexander Pivovarov wrote:
>
>    I think EMR has its own limitation
>
> e.g. I want to setup hadoop 2.6.0 with kerberos + hive-1.2.0 to test my
> hive patch.
>  How EMR can help me?  it supports hadoop up to 2.4.0  (not even 2.4.1)
>
> http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-hadoop-version.html
>
>
>
>
>
>
> On Thu, Mar 5, 2015 at 9:51 PM, Jonathan Aquilina <jaquilina@eagleeyet.net
> > wrote:
>
>>  Hi guys I know you guys want to keep costs down, but why go through all
>> the effort to setup ec2 instances when you deploy EMR it takes the time to
>> provision and setup the ec2 instances for you. All configuration then for
>> the entire cluster is done on the master node of the particular cluster or
>> setting up of additional software that is all done through the EMR console.
>> We were doing some geospatial calculations and we loaded a 3rd party jar
>> file called esri into the EMR cluster. I then had to pass a small bootstrap
>> action (script) to have it distribute esri to the entire cluster.
>>
>> Why are you guys reinventing the wheel?
>>
>>
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>>   On 2015-03-06 03:35, Alexander Pivovarov wrote:
>>
>>    I found the following solution to this problem
>>
>> I registered 2 subdomains  (public and local) for each computer on
>> https://freedns.afraid.org/subdomain/
>> e.g.
>> myhadoop-nn.crabdance.com
>> myhadoop-nn-local.crabdance.com
>>
>> then I added cron job which sends http requests to update public and
>> local ip on freedns server
>> hint: public ip is detected automatically
>> ip address for local name can be set using request parameter &address=10.x.x.x
>> (don't forget to escape &)
>>
>> as a result my nn computer has 2 DNS names with currently assigned ip
>> addresses , e.g.
>> myhadoop-nn.crabdance.com  54.203.181.177
>> myhadoop-nn-local.crabdance.com   10.220.149.103
>>
>> in hadoop configuration I can use local machine names
>> to access my cluster outside of AWS I can use public names
>>
>> Just curious if AWS provides easier way to name EC2 computers?
>>
>> On Thu, Mar 5, 2015 at 5:19 PM, Jonathan Aquilina <
>> jaquilina@eagleeyet.net> wrote:
>>
>>>  I dont know how you would do that to be honest. With EMR you have
>>> destinctions master core and task nodes. If you need to change
>>> configuration you just ssh into the EMR master node.
>>>
>>>
>>>
>>> ---
>>> Regards,
>>> Jonathan Aquilina
>>> Founder Eagle Eye T
>>>
>>>   On 2015-03-06 02:11, Alexander Pivovarov wrote:
>>>
>>> What is the easiest way to assign names to aws ec2 computers?
>>> I guess computer need static hostname and dns name before it can be used
>>> in hadoop cluster.
>>> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <jaquilina@eagleeyet.net>
>>> wrote:
>>>
>>>>  When I started with EMR it was alot of testing and trial and error.
>>>> HUE is already supported as something that can be installed from the AWS
>>>> console. What I need to know is if you need this cluster on all the time
or
>>>> this is goign ot be what amazon call a transient cluster. Meaning you fire
>>>> it up run the job and tear it back down.
>>>>
>>>>
>>>>
>>>> ---
>>>> Regards,
>>>> Jonathan Aquilina
>>>> Founder Eagle Eye T
>>>>
>>>>  On 2015-03-06 01:10, Krish Donald wrote:
>>>>
>>>>  Thanks Jonathan,
>>>>
>>>> I will try to explore EMR option also.
>>>> Can you please let me know the configuration which you have used it?
>>>> Can you please recommend for me also?
>>>> I would like to setup Hadoop cluster using cloudera manager and then
>>>> would like to do below things:
>>>>
>>>> setup kerberos
>>>> setup federation
>>>> setup monitoring
>>>> setup hadr
>>>> backup and recovery
>>>> authorization using sentry
>>>> backup and recovery of individual componenets
>>>> performamce tuning
>>>> upgrade of cdh
>>>> upgrade of CM
>>>> Hue User Administration
>>>> Spark
>>>> Solr
>>>>
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>>
>>>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina <
>>>> jaquilina@eagleeyet.net> wrote:
>>>>
>>>>>  krish EMR wont cost you much with all the testing and data we ran
>>>>> through the test systems as well as the large amont of data when everythign
>>>>> was read we paid about 15.00 USD. I honestly do not think that the specs
>>>>> there would be enough as java can be pretty ram hungry.
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Regards,
>>>>> Jonathan Aquilina
>>>>> Founder Eagle Eye T
>>>>>
>>>>>   On 2015-03-06 00:41, Krish Donald wrote:
>>>>>
>>>>>  Hi,
>>>>>
>>>>> I am new to AWS and would like to setup Hadoop cluster using cloudera
>>>>> manager for 6-7 nodes.
>>>>>
>>>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
>>>>> I would like to use free service as of now.
>>>>>
>>>>> Please advise.
>>>>>
>>>>> Thanks
>>>>> Krish
>>>>>
>>>>>

Mime
View raw message