hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: EC2 scripts
Date Thu, 19 Nov 2009 23:27:07 GMT
Thanks Naresh. Thanks for taking the time to try this out. I will incorporate your suggestions.


> error:  "fs.epoll.max_user_instance"  is an unknown key

This is a bit of future proofing. That's not a known sysctl key until kernel 2.6.27, at which
point oddly low epoll user descriptor limits go into effect. See http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/.
At some point there may be a 2.6.27 based AKI. I could /dev/null the message but then some
other more serious potential problem with sysctl would be hidden. 

> Also shouldnt you be having   core-site.xml and hdfs-site.xml  as it is  hadoop-0.20.1


Yes. What I did for this initial work is adapt the Hadoop EC2 scripts, which target 0.19.


   - Andy




________________________________
From: Naresh Rapolu <nrapolu@purdue.edu>
To: hbase-user@hadoop.apache.org
Sent: Thu, November 19, 2009 2:52:18 PM
Subject: Re: EC2  scripts

Thanks Andrew,

Your scripts are working fine.  We restarted everything and  tested, and they are working
fine.  A few issues though :
-  While starting,   launch-hbase-cluster  gives the following  error.
   error:  "fs.epoll.max_user_instance"  is an unknown key.    It occurs during  starting
zookeeper instances.  
-  We needed MapReduce along with HBase.  The note on the JIRA page that you only need to
add only two lines in hbase-ec2-env.sh    is insufficient.
  The following changes need to be made.
  1. hbase-ec2-env.sh   should write  mapred.job.tracker   property into  hadoop-site.xml
  (  Also shouldnt you be having   core-site.xml and hdfs-site.xml  as it is  hadoop-0.20.1
???   Infact because of this , there are warning messages all over the place when you are
using   hdfs  through command line ).
  2.   HADOOP_CLASSPATH  in   hadoop/conf/hadoop-env.sh   needs to be changed in the underlying
 AMI,  to include  hbase, zookeeper jars and conf directory.    Probably you can modify the
public AMI, and recreate the bundle  as the  paths to these are known apriori.   3.  For other
users,  the following three lines should be added in  hbase-ec2-env.sh
       For master:
       "$HADOOP_HOME"/bin/hadoop-daemon.sh start jobtracker
       "$HADOOP_HOME"/bin/hadoop-daemon.sh start tasktracker

       For slave:
       "$HADOOP_HOME"/bin/hadoop-daemon.sh start tasktracker.

Thanks Andrew, for the scripts and your feedback.. They were very timely.
Naresh Rapolu.


Andrew Purtell wrote:
> I just performed the following steps:
> 
> 1) Install pristine RHEL 5 (well, CentOS) system in VM.
> 
> 2) Download and unzip the EC2 AMI and API tools zipfiles.
> 
> 3) Put the tools on the path and set EC2_HOME in the environment.
> 
> 4) Pristine checkout of HBase trunk.
> 
> 5) Configure hbase-ec2-env.sh 
> 6) ./bin/hbase-ec2 launch-cluster testcluster 3 3
> 
> 7) ./bin/hbase-ec2 login testcluster
> 
> 8) hbase shell
> 
> 9) status 'simple'
> 
> 3 live servers
>     domU-12-31-39-09-75-11.compute-1.internal:60020 1258653694915
>         requests=0, regions=1, usedHeap=29, maxHeap=987
>     domU-12-31-39-01-AC-31.compute-1.internal:60020 1258653709041
>         requests=0, regions=1, usedHeap=29, maxHeap=987
>     domU-12-31-39-01-B0-91.compute-1.internal:60020 1258653706411
>         requests=0, regions=0, usedHeap=27, maxHeap=987
> 0 dead servers
> 
> I would be very interested to see what is in the HBase logs on your slaves, and also
the full output of the cluster launch if you can capture that into a buffer and save it to
a file. 
> Thanks,
> 
>     - Andy
> 
> 
> 
> 
> 
> ________________________________
> From: Andrew Purtell <apurtell@apache.org>
> To: hbase-user@hadoop.apache.org
> Sent: Thu, November 19, 2009 9:28:51 AM
> Subject: Re: EC2  scripts
> 
> So it seems the slaves are not checking in with the master for some reason. 
> Can you provide the log content from the slave instances in /mnt/hbase/logs/*.log ? 
> What is the output of 'hadoop dfsadmin -report'? 
> Thanks for trying this out Naresh. 
>    - Andy
> 
> 
> 
> 
> ________________________________
> From: Naresh Rapolu <nrapolu@purdue.edu>
> To: hbase-user@hadoop.apache.org
> Sent: Thu, November 19, 2009 9:09:54 AM
> Subject: Re: EC2  scripts
> 
> Thanks Andrew,
> 
> We`ve made a private instance with our keys,  by altering your public hbase AMI and used
it to launch the cluster instances.
> We are now able to do  "ssh slave" from master.  But, still the problem persists.  From
the hbase shell of master,  "status 'simple' " returns  0 live servers.
> Also, while creating a new table from the hbase shell, the following  error pops up.
> 
> hbase(main):001:0> create 'dummy', 'dummyfamily'
> NativeException: org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException
>        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:750)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 
>        from sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
>        from sun/reflect/NativeConstructorAccessorImpl.java:39:in `newInstance'
>        from sun/reflect/DelegatingConstructorAccessorImpl.java:27:in `newInstance'
>        from java/lang/reflect/Constructor.java:513:in `newInstance'
>        from org/apache/hadoop/hbase/RemoteExceptionHandler.java:94:in `decodeRemoteException'
>        from org/apache/hadoop/hbase/client/HBaseAdmin.java:208:in `createTableAsync'
>        from org/apache/hadoop/hbase/client/HBaseAdmin.java:165:in `createTable'
>       -- We`ve checked and ensured that all slaves are online with  HRegionServer  daemons.
Btw,  Hadoop is working well,  we are able to put in large files into the dfs.   Problem seems
to be with Hbase alone.
> 
> Let me know if you need any more log messages.
> 
> Thanks,
> Naresh Rapolu.
> 
> Andrew Purtell wrote:
>  
>> 1) How long did you wait before logging on to the master and checking status? I bet
if you checked the slave instances status at that time (e.g. using elasticfox) they were still
pending startup. The slaves are started last and EC2 may not start them right away nor all
at once.
>> 
>> 2) The way private keys work with EC2 instances you must supply the EC2 private key
using -i unless you also use this as your default private key, set up the appropriate config
(man 5 ssh_config), or build your own private instance with your default key already installed.
On Wed Nov 18th, 2009 9:49 PM PST Naresh Rapolu wrote:
>> 
>>      
>>> Hello ,
>>> 
>>> We were trying to use the  src/contrib/ec2  scripts.  All the instances started
well,  but from  hbase shell, on the master ,  a   " status 'simple' "   listed  0  live servers.
Also , we are not able to ssh into any of the slave instances from the master,  without using
 " shh -i  private_key.pem  ".   (ie)  ssh is not configured to be used without a password
and with automatic keys.   I guess, the instances still maintain the creator`s private key.
  Did they work for any others, apart from the creator ?   Even the  JIRA page shows that
 it didnt completely work  for  St.Ack.
>>> 
>>> Thanks,
>>> Naresh Rapolu
>>>          
>> 
>>            
> 
> 
>        


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message