accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ke...@deenlo.com
Subject Re: Review Request 23397: ACCUMULO-2984 Support Running MAC against a standard instance
Date Tue, 21 Oct 2014 15:09:53 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23397/#review57582
-----------------------------------------------------------



minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
<https://reviews.apache.org/r/23397/#comment98367>

    I looked into this code, its full of landmines.  It calls VolumeManager.get() which uses
static config for accumulo-site.xml (which may not be the accumulo-site you want).   Want
to avoid this.  Want to make sure that all info is obtained using your accumulo-site.xml and
your haddop config.
    
    Below I put together some code that I think mostly does this.  Tried to make sure the
instance id is obtained from filesystem using your config and that zookeeper is obtained using
your config.
    
    ```java
    Configuration config = new Configuration(false);
    config.addResource(yourAccumuloSite);
    
    ConfigurationCopy cc = new ConfigurationCopy(config);
    try {
      fs = VolumeManagerImpl.get(cc);
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
    Path instanceIdPath = Accumulo.getAccumuloInstanceIdPath(fs);
    
    //TODO this method calls CahcedConfig.getInstance(), should create a method where hadoop
config is passed in
    String instanceIdFromFile = ZooUtil.getInstanceIDFromHdfs(instanceIdPath, cc);
    
    zrw = new ZooReaderWriterFactory.getZooReaderWriter(cc.get(Property.INSTANCE_ZK_HOST),
(int) cc.getTimeInMillis(Property.INSTANCE_ZK_TIMEOUT),
                cc.get(Property.INSTANCE_SECRET));
    
    String rootPath = ZooUtil.getRoot(instanceIdFromFile);
    //TODO pass this root path to isAccumuloOffline
    
    ```


- kturner


On Oct. 21, 2014, 12:32 a.m., John Vines wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23397/
> -----------------------------------------------------------
> 
> (Updated Oct. 21, 2014, 12:32 a.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-2984
>     https://issues.apache.org/jira/browse/ACCUMULO-2984
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> Adds a change to SiteConfiguration to allow external setting of the xml configuration
file.
> Adds a single method to MiniAccumuloConfig which allows a user to point to accumulo-site.xml
and HADOOP_CONF_DIR to use for pulling out requisite instance information
> Clusters configurations into those required to run inside a MAC-sized footprint and those
which are for arbitrary naming schemes for MAC
> Provides flagging to prevent uneccessary folder creation
> Provides flagging to prevent running zookeeper and initializing
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/accumulo/core/conf/SiteConfiguration.java 9b65e7d 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/MiniAccumuloConfig.java be80f85

>   minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
5d8501e 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java
e9ad045 
>   server/base/src/main/java/org/apache/accumulo/server/client/HdfsZooInstance.java 3508164

>   server/base/src/main/java/org/apache/accumulo/server/conf/ServerConfiguration.java
50dec57 
>   server/base/src/main/java/org/apache/accumulo/server/util/AccumuloStatus.java PRE-CREATION

>   server/base/src/main/java/org/apache/accumulo/server/zookeeper/ZooReaderWriter.java
435591d 
>   test/src/test/java/org/apache/accumulo/test/ExistingMacIT.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/23397/diff/
> 
> 
> Testing
> -------
> 
> Ran the following test code-
> public class TestMACWithRealInstance {
>   public static void main(String args[]) throws IOException, AccumuloException, AccumuloSecurityException,
TableExistsException, InterruptedException {
>     MiniAccumuloConfig macConfig = new MiniAccumuloConfig(new File("/tmp/mac"), "secret");
>     macConfig.setNumTservers(2);
>     macConfig.setMemory(ServerType.TABLET_SERVER, 2, MemoryUnit.GIGABYTE);
>     macConfig.useExistingInstance(new File("/usr/lib/accumulo/conf/accumulo-site.xml"),
new File("/usr/lib/hadoop/conf"));
>     MiniAccumuloCluster mac = new MiniAccumuloCluster(macConfig);
>     mac.start();
>     System.out.println("Started");
>     mac.getConnector("root", "secret").tableOperations().create("macCreated");
>     System.out.println("Stopping");
>     mac.stop();
>     System.out.println("Stopped");
>   }
> }
> Which runs fine, except stopping issues which seem to be related to ACCUMULO-2985
> 
> After running this, I validated that the table was created in the real accumulo instance
via zkCli
> 
> 
> Thanks,
> 
> John Vines
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message