hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: hbase on ec2 with s3 anyone?
Date Wed, 07 May 2008 22:22:29 GMT
sorry if I wasn't clear that the new scripts and old scripts were not  
compatible. thus their being in 0.17.0, not 0.16.4.

you would need to build new images anyway since you need HBase  
installed and started at boot time.


On May 7, 2008, at 3:15 PM, Jim R. Wilson wrote:

> I've come to the conclusion that using the contrib/ec2 scripts from
> hadoop 0.17 is incompatible with the prebuilt hadoop-0.16.1 image
> currently available in the hadoop-ec2-images bucket (ami-461df82f
> hadoop-ec2-images/hadoop-0.16.1.manifest.xml to be precise).
>
> The problem is that the user-data passed in by 0.17 has a different
> format than what is expected by the hadoop-init script packaged with
> 0.16.1.  Specifically, 0.17's user -data is meant to be a comma
> delimited list of bash var settings of the form KEY=VAL, whereas
> 0.16.x seems to expect just a comma delimited list of values whose
> keys are known by their ordinal placement (that is, the first value is
> the number of instances, the second value is the name of the master
> node).
>
> So now I'm back to the idea that I'm going to have to build myself an
> ec2 AMI with hadoop 0.17 from "scratch" (using the create-instance
> scripts of course).  This isn't /too/ much more work than I'd have to
> do anyway.  I plan on running hbase on my cluster as well as python
> hadoop-streaming jobs which was going to require other libraries (like
> SQLAlchemy and Thrift).  These items were going to necessitate
> creating my own images anyway :/
>
> -- Jim
>
> On Wed, May 7, 2008 at 3:51 PM, Jim R. Wilson  
> <wilson.jim.r@gmail.com> wrote:
>>> keep the questions coming. will be glad to see HBase running on  
>>> ec2, maybe
>>> we can put your changes back into the tree.
>>
>> Thanks :)
>>
>> Just prior to this excursion, I found some things needing tweaking to
>> work with my hosting provider.  Now that I'm moving to ec2, I'll be  
>> on
>> the lookout for similar issues.  I'll submit any patches I end up
>> making.
>>
>> -- Jim
>>
>>
>>
>> On Wed, May 7, 2008 at 3:39 PM, Chris K Wensel <chris@wensel.net>  
>> wrote:
>>>
>>>> In the aforementioned EC2 wiki page, it has a configuration section
>>>> labeled "(Pre 0.17) Hadoop cluster variables (GROUP, MASTER_HOST,
>>>> NO_INSTANCES)".  I had assumed that I would actually need  
>>>> hadoop-0.17
>>>> on my ec2 instances in order to forgo those instructions.  It  
>>>> sounds
>>>> like you're telling me that all the "pre 0.17" or "0.17" specific
>>>> instructions in the wiki page refer only to the ec2 creation  
>>>> scripts,
>>>> not the actual running version in the cluster, is that correct?
>>>>
>>>>
>>>
>>> correct.
>>>
>>> the only reason these scripts are in the 0.17 branch is because  
>>> they are
>>> not backward compatible with themselves.
>>>
>>>
>>>
>>>> Also, how safe is it to run a different version of the ec2 scripts
>>>> from the actual running hadoop instance?  I'm guessing it's pretty
>>>> safe since you suggested it :)
>>>>
>>>>
>>>
>>> there are no dependencies between the EC2 scripts and Hadoop core.  
>>> you can
>>> use them with any version, as long as you build EC2 Images for the  
>>> versions
>>> of Hadoop you are after with the 'new' scripts.
>>>
>>>
>>>
>>>> Thanks again for all the help - still wrapping my mind around  
>>>> this stuff.
>>>>
>>>>
>>>
>>> keep the questions coming. will be glad to see HBase running on  
>>> ec2, maybe
>>> we can put your changes back into the tree.
>>>
>>>
>>>
>>> Chris K Wensel
>>> chris@wensel.net
>>> http://chris.wensel.net/
>>> http://www.cascading.org/
>>>
>>>
>>>
>>>
>>>
>>

Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
http://www.cascading.org/





Mime
View raw message