hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guilherme Germoglio <germog...@gmail.com>
Subject Re: How to initialize a test environment ?
Date Tue, 13 Oct 2009 18:35:28 GMT
Hello Guillaume,
What about extending org.apache.hadoop.hbase.HBaseClusterTestCase,
instantiating it and invoking its setUp()? I know it isn't the most
beautiful way to start HBase from java, but it works pretty well here.

Something along the lines of:

public class StandAloneServer extends HBaseClusterTestCase {

    public StandAloneServer(Properties props) {
        super();
        // iterate over props map invoking: conf.set(propKey, propValue);
    }

    public static void main(String[] args) {
      Properties props;
      // load props from args[0], for example

      StandAloneServer sas = new StandAloneServer(props);
       // add close hook
      Runtime.getRuntime().addShutdownHook(new Thread() {
         public void run() {
             sas.tearDown();
         }
      });

      sas.setUp();
    }

}

2009/10/13 Guillaume Dufrêne <dufrene@argia.fr>

> stack wrote:
>
>> A few comments interspersed in the below:
>>
>> 2009/10/13 Guillaume Dufrêne <dufrene@argia.fr>
>>
>>
>>
>>
>>> I wrote a class inspired from the Mini*Server located in Hbase Tests.
>>>
>>> (I use Hbase 0.20)
>>>
>>> Here my class :
>>>
>>> public class StandaloneServer {
>>>
>>>  private static Log log = LogFactory.getLog(StandaloneServer.class);
>>>
>>>  private MiniZooKeeperCluster zookeeper;
>>>  private MiniHBaseCluster hbase;
>>>  private MiniDFSCluster dfs;
>>>
>>>  private HBaseConfiguration conf;
>>>
>>>  public void startup() throws Exception {
>>>     conf = new HBaseConfiguration();
>>>
>>>     log.info("Start DFS ...");
>>>     dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
>>>     // set file system to the mini dfs just started up
>>>     FileSystem fs = dfs.getFileSystem();
>>>     conf.set("fs.default.name", fs.getUri().toString());
>>>
>>>
>>>
>>
>> You are using 0.20?  FYI, the above name is deprecated in TRUNK.  HBase
>> TRUNK has been changed to use new name.
>>
>>
>>
>>
> Yep, I use 0.20 release, I didn't build hbase myself from the trunk.
> It seems that "fs.default.name" is used in the 0.20 release as shown in
> unit tests.
>
>
>      Path parentdir = fs.getHomeDirectory();
>>>     conf.set(HConstants.HBASE_DIR, parentdir.toString());
>>>     fs.mkdirs(parentdir);
>>>     FSUtils.setVersion(fs, parentdir);
>>>
>>>     log.info("Start Zookeeper ...");
>>>     zookeeper = new MiniZooKeeperCluster();
>>>     int clientPort = zookeeper.startup(new File(parentdir.toString(),
>>> "zookeeper"));
>>>     conf.set("hbase.zookeeper.property.clientPort", Integer
>>>             .toString(clientPort));
>>>         // Disable regionserver info.
>>>     conf.set("hbase.regionserver.info.port", "-1");
>>>     conf.set("hbase.master.info.port", "-1");
>>>
>>>     log.info("Start HBase ...");
>>>     hbase = new MiniHBaseCluster(conf, 1);
>>>
>>>     log.info("Check ROOT metatable connexion ...");
>>>     new HTable(conf, HConstants.META_TABLE_NAME);
>>>
>>>  }
>>>
>>>  public void shutdown() {
>>>     hbase.shutdown();
>>>     if (dfs != null) {
>>>         try {
>>>             FileSystem fs = dfs.getFileSystem();
>>>             if (fs != null)
>>>                 fs.close();
>>>         } catch (IOException e) {
>>>             System.err.println("error closing file system: " + e);
>>>         }
>>>         try {
>>>             dfs.shutdown();
>>>         } catch (Exception e) { /* ignore */
>>>         }
>>>     }
>>>  }
>>>
>>>
>>>
>>
>> The shutdown thread stuff is different in TRUNK.
>>
>>
>
> Thanks for that information, I will update my code when I migrate hbase
> version.
>
>
>   public static void main(String[] args) throws Exception {
>>>     final StandaloneServer server = new StandaloneServer();
>>>     server.startup();
>>>
>>>     // add close hook
>>>     Runtime.getRuntime().addShutdownHook(new Thread() {
>>>         public void run() {
>>>             server.shutdown();
>>>         }
>>>     });
>>>
>>>  }
>>> }
>>>
>>> I first wrote a main operation to easily test the behaviour ...
>>> The idea is to start the server when my unit test is set up.
>>>
>>> I got the following error :
>>> 09/10/13 14:12:13 INFO hbase.StandaloneServer: Check ROOT metatable
>>> connexion ...
>>> Exception in thread "main"
>>> org.apache.hadoop.hbase.client.NoServerForRegionException: No server
>>> address
>>> listed in -ROOT- for region .META.,,1
>>>  at
>>>
>>> [oahh.client]HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:660)
>>>
>>>  at
>>>
>>> [oahh.client].HConnectionManager$TableServers.locateRegion(HConnectionManager.java:582)
>>>
>>>  at
>>>
>>> [oahh.client].client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:549)
>>>
>>>  at [oahh.client].HTable.<init>(HTable.java:125)
>>>  at (...) StandaloneServer.startup(StandaloneServer.java:53)
>>>  at (...) StandaloneServer.main(StandaloneServer.java:76)
>>>
>>>
>>>
>>
>>
>> This happens on start up until root is deployed... you'll see one or two
>> in
>> the logs.
>>
>>
> The StackTrace is triggered from my last line of code that check access to
> META table.
> The MiniHBaseCluster is supposed to wait for region server to start, isn't
> it ?
> Does the "root" you pointed out be the HBase master ? (I suppose so)
> What should I do to ensure this root deployment ?
> do a try/catch and retry N times with a delay ?
>
>  And some errors about "NO JSP Support" (but I think that I can ignore
>>> them,
>>> no ?)
>>>
>>>
>>>
>> You've turned off the info servers -- the web uis -- so should be fine.
>>
>>
>>
>>
>>
>>> So my question are :
>>> - Is it the good way to start a test environment ?
>>>
>>>
>>>
>>
>> Yeah, you seem to have figured out our ugly test setup.  You could also
>> have
>> run hbase with all defaults?  Then it runs a master and regionserver in
>> same
>> in JVM writing the local fileystem.
>>
>>
> You mean that I can run "LocalHBaseCluster" with a new HBaseConfiguration()
> without starting FS ?
>
> If Someone has a java code sample that start a hbase, let's share ;-)
>
>> [...]
>>>
>>
> Thanks.
> --
> Guillaume.
>
>
>
>
>
>
> Ce message et toutes les pièces jointes sont établis à l'attention
> exclusive de leurs destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le détruire et d'en avertir immédiatement
> l'expéditeur. L'internet ne permettant pas d'assurer l'intégrité de ce
> message, le contenu de ce message ne représente en aucun cas un engagement
> de la part de Leroy Merlin.
>
>


-- 
Guilherme

msn: guigermoglio@hotmail.com
homepage: http://sites.google.com/site/germoglio/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message