hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Pivovarov <apivova...@gmail.com>
Subject Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
Date Fri, 06 Mar 2015 06:31:15 GMT
I think EMR has its own limitation

e.g. I want to setup hadoop 2.6.0 with kerberos + hive-1.2.0 to test my
hive patch.

How EMR can help me?  it supports hadoop up to 2.4.0  (not even 2.4.1)
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-hadoop-version.html







On Thu, Mar 5, 2015 at 9:51 PM, Jonathan Aquilina <jaquilina@eagleeyet.net>
wrote:

>  Hi guys I know you guys want to keep costs down, but why go through all
> the effort to setup ec2 instances when you deploy EMR it takes the time to
> provision and setup the ec2 instances for you. All configuration then for
> the entire cluster is done on the master node of the particular cluster or
> setting up of additional software that is all done through the EMR console.
> We were doing some geospatial calculations and we loaded a 3rd party jar
> file called esri into the EMR cluster. I then had to pass a small bootstrap
> action (script) to have it distribute esri to the entire cluster.
>
> Why are you guys reinventing the wheel?
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
>  On 2015-03-06 03:35, Alexander Pivovarov wrote:
>
>    I found the following solution to this problem
>
> I registered 2 subdomains  (public and local) for each computer on
> https://freedns.afraid.org/subdomain/
> e.g.
> myhadoop-nn.crabdance.com
> myhadoop-nn-local.crabdance.com
>
> then I added cron job which sends http requests to update public and local
> ip on freedns server
> hint: public ip is detected automatically
> ip address for local name can be set using request parameter &address=10.x.x.x
> (don't forget to escape &)
>
> as a result my nn computer has 2 DNS names with currently assigned ip
> addresses , e.g.
> myhadoop-nn.crabdance.com  54.203.181.177
> myhadoop-nn-local.crabdance.com   10.220.149.103
>
> in hadoop configuration I can use local machine names
> to access my cluster outside of AWS I can use public names
>
> Just curious if AWS provides easier way to name EC2 computers?
>
> On Thu, Mar 5, 2015 at 5:19 PM, Jonathan Aquilina <jaquilina@eagleeyet.net
> > wrote:
>
>>  I dont know how you would do that to be honest. With EMR you have
>> destinctions master core and task nodes. If you need to change
>> configuration you just ssh into the EMR master node.
>>
>>
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>>   On 2015-03-06 02:11, Alexander Pivovarov wrote:
>>
>> What is the easiest way to assign names to aws ec2 computers?
>> I guess computer need static hostname and dns name before it can be used
>> in hadoop cluster.
>> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <jaquilina@eagleeyet.net>
>> wrote:
>>
>>>  When I started with EMR it was alot of testing and trial and error.
>>> HUE is already supported as something that can be installed from the AWS
>>> console. What I need to know is if you need this cluster on all the time or
>>> this is goign ot be what amazon call a transient cluster. Meaning you fire
>>> it up run the job and tear it back down.
>>>
>>>
>>>
>>> ---
>>> Regards,
>>> Jonathan Aquilina
>>> Founder Eagle Eye T
>>>
>>>  On 2015-03-06 01:10, Krish Donald wrote:
>>>
>>>  Thanks Jonathan,
>>>
>>> I will try to explore EMR option also.
>>> Can you please let me know the configuration which you have used it?
>>> Can you please recommend for me also?
>>> I would like to setup Hadoop cluster using cloudera manager and then
>>> would like to do below things:
>>>
>>> setup kerberos
>>> setup federation
>>> setup monitoring
>>> setup hadr
>>> backup and recovery
>>> authorization using sentry
>>> backup and recovery of individual componenets
>>> performamce tuning
>>> upgrade of cdh
>>> upgrade of CM
>>> Hue User Administration
>>> Spark
>>> Solr
>>>
>>>
>>> Thanks
>>> Krish
>>>
>>>
>>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina <
>>> jaquilina@eagleeyet.net> wrote:
>>>
>>>>  krish EMR wont cost you much with all the testing and data we ran
>>>> through the test systems as well as the large amont of data when everythign
>>>> was read we paid about 15.00 USD. I honestly do not think that the specs
>>>> there would be enough as java can be pretty ram hungry.
>>>>
>>>>
>>>>
>>>> ---
>>>> Regards,
>>>> Jonathan Aquilina
>>>> Founder Eagle Eye T
>>>>
>>>>   On 2015-03-06 00:41, Krish Donald wrote:
>>>>
>>>>  Hi,
>>>>
>>>> I am new to AWS and would like to setup Hadoop cluster using cloudera
>>>> manager for 6-7 nodes.
>>>>
>>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
>>>> I would like to use free service as of now.
>>>>
>>>> Please advise.
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>>

Mime
View raw message