Return-Path: X-Original-To: apmail-accumulo-commits-archive@www.apache.org Delivered-To: apmail-accumulo-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 31DD59906 for ; Thu, 3 Jan 2013 19:18:32 +0000 (UTC) Received: (qmail 89319 invoked by uid 500); 3 Jan 2013 19:18:32 -0000 Delivered-To: apmail-accumulo-commits-archive@accumulo.apache.org Received: (qmail 89292 invoked by uid 500); 3 Jan 2013 19:18:32 -0000 Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 89285 invoked by uid 99); 3 Jan 2013 19:18:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2013 19:18:32 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2013 19:18:27 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id DFFA3238896F; Thu, 3 Jan 2013 19:18:07 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1428556 - in /accumulo/trunk: ./ bin/ core/src/main/java/org/apache/accumulo/core/conf/ core/src/main/java/org/apache/accumulo/core/master/thrift/ server/src/main/java/org/apache/accumulo/server/tabletserver/ start/ start/src/main/java/org... Date: Thu, 03 Jan 2013 19:18:06 -0000 To: commits@accumulo.apache.org From: kturner@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20130103191807.DFFA3238896F@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: kturner Date: Thu Jan 3 19:18:05 2013 New Revision: 1428556 URL: http://svn.apache.org/viewvc?rev=1428556&view=rev Log: ACCUMULO-708 ACCUMULO-866 ACCUMULO-869 ACCUMULO-870 many changes to accumulo-start Added: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ - copied from r1427869, accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ - copied from r1428462, accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider/ Removed: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloFilesystemAlterationMonitor.java accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/Test.java accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/ accumulo/trunk/start/src/test/resources/application1/ Modified: accumulo/trunk/bin/accumulo accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java accumulo/trunk/pom.xml accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java accumulo/trunk/start/pom.xml accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml Modified: accumulo/trunk/bin/accumulo URL: http://svn.apache.org/viewvc/accumulo/trunk/bin/accumulo?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/bin/accumulo (original) +++ accumulo/trunk/bin/accumulo Thu Jan 3 19:18:05 2013 @@ -48,11 +48,7 @@ bin=`cd "$bin"; pwd` . "$bin"/config.sh -START_JAR=$ACCUMULO_HOME/lib/accumulo-start-$ACCUMULO_VERSION.jar -COMMONS_JCI_JARS=$ACCUMULO_HOME/lib/commons-jci-core-1.0.jar:$ACCUMULO_HOME/lib/commons-jci-fam-1.0.jar:$ACCUMULO_HOME/lib/log4j-1.2.16.jar:$ACCUMULO_HOME/lib/commons-logging-1.0.4.jar:$ACCUMULO_HOME/lib/commons-logging-api-1.0.4.jar -START_CLASSES_DIR=$ACCUMULO_HOME/src/start/target/classes -COMMONS_VFS_JARS=$ACCUMULO_HOME/lib/commons-vfs2-2.0.jar -HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath` +START_JAR=$ACCUMULO_HOME/lib/accumulo-start-$ACCUMULO_VERSION.jar:$ACCUMULO_HOME/lib/log4j-1.2.16.jar locationByProgram() { @@ -95,7 +91,7 @@ else fi XML_FILES=${ACCUMULO_HOME}/conf -CLASSPATH=${XML_FILES}:${START_JAR}:${COMMONS_JCI_JARS}:${COMMONS_VFS_JARS}:${HADOOP_CLASSPATH} +CLASSPATH=${XML_FILES}:${START_JAR} if [ -z $JAVA_HOME -o ! -d $JAVA_HOME ]; then echo "JAVA_HOME is not set. Please make sure it's set globally or in conf/accumulo-env.sh" Modified: accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java URL: http://svn.apache.org/viewvc/accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java (original) +++ accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java Thu Jan 3 19:18:05 2013 @@ -24,6 +24,7 @@ import org.apache.accumulo.core.file.rfi import org.apache.accumulo.core.util.format.DefaultFormatter; import org.apache.accumulo.core.util.interpret.DefaultScanInterpreter; import org.apache.accumulo.start.classloader.AccumuloClassLoader; +import org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader; public enum Property { // instance properties (must be the same for every node in an instance) @@ -44,11 +45,12 @@ public enum Property { // general properties GENERAL_PREFIX("general.", null, PropertyType.PREFIX, "Properties in this category affect the behavior of accumulo overall, but do not have to be consistent throughout a cloud."), - GENERAL_CLASSPATHS(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.DEFAULT_CLASSPATH_VALUE, PropertyType.STRING, + GENERAL_CLASSPATHS(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.ACCUMULO_CLASSPATH_VALUE, PropertyType.STRING, "A list of all of the places to look for a class. Order does matter, as it will look for the jar " + "starting in the first location to the last. Please note, hadoop conf and hadoop lib directories NEED to be here, " + "along with accumulo lib and zookeeper directory. Supports full regex on filename alone."), // needs special treatment in accumulo start jar - GENERAL_DYNAMIC_CLASSPATHS(AccumuloClassLoader.DYNAMIC_CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.DEFAULT_DYNAMIC_CLASSPATH_VALUE, PropertyType.STRING, + GENERAL_DYNAMIC_CLASSPATHS(AccumuloVFSClassLoader.DYNAMIC_CLASSPATH_PROPERTY_NAME, AccumuloVFSClassLoader.DEFAULT_DYNAMIC_CLASSPATH_VALUE, + PropertyType.STRING, "A list of all of the places where changes in jars or classes will force a reload of the classloader."), GENERAL_RPC_TIMEOUT("general.rpc.timeout", "120s", PropertyType.TIMEDURATION, "Time to wait on I/O for simple, short RPC calls"), GENERAL_KERBEROS_KEYTAB("general.kerberos.keytab", "", PropertyType.PATH, "Path to the kerberos keytab to use. Leave blank if not using kerberoized hdfs"), @@ -295,12 +297,9 @@ public enum Property { //VFS ClassLoader properties - VFS_CLASSLOADER_PREFIX("classloader.vfs", null, PropertyType.PREFIX, "Properties in this category affect the VFS ClassLoader"), - VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY("classloader.vfs.context.classpath.system", "", PropertyType.STRING, - "Classpath for the system context"), - VFS_CLASSLOADER_CONTEXT_NAMES_PROPERTY("classloader.vfs.context.names", "", PropertyType.STRING, - "Comma separated list of context names"), - VFS_CONTEXT_CLASSPATH_PROPERTY("classloader.vfs.context.classpath.", null, PropertyType.PREFIX, "Classpath for this context"); + VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY(AccumuloVFSClassLoader.VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY, "", PropertyType.STRING, + "Configuration for a system level vfs classloader. Accumulo jar can be configured here and loaded out of HDFS."), + VFS_CONTEXT_CLASSPATH_PROPERTY(AccumuloVFSClassLoader.VFS_CONTEXT_CLASSPATH_PROPERTY, null, PropertyType.PREFIX, "Classpath for this context"); Modified: accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java URL: http://svn.apache.org/viewvc/accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java (original) +++ accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java Thu Jan 3 19:18:05 2013 @@ -1111,7 +1111,7 @@ import org.slf4j.LoggerFactory; for (int _i1 = 0; _i1 < _map0.size; ++_i1) { String _key2; // required - TableInfo _val3; // optional + TableInfo _val3; // required _key2 = iprot.readString(); _val3 = new TableInfo(); _val3.read(iprot); Modified: accumulo/trunk/pom.xml URL: http://svn.apache.org/viewvc/accumulo/trunk/pom.xml?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/pom.xml (original) +++ accumulo/trunk/pom.xml Thu Jan 3 19:18:05 2013 @@ -144,7 +144,7 @@ ../lib - commons-collections,commons-configuration,commons-io,commons-lang,jline,log4j,libthrift,commons-jci-core,commons-jci-fam,commons-logging,commons-logging-api,commons-vfs2,gson,jcommander + commons-collections,commons-configuration,commons-io,commons-lang,jline,log4j,libthrift,commons-logging,commons-logging-api,commons-vfs2,gson,jcommander true @@ -195,11 +195,6 @@ maven-surefire-plugin - - - .. - - Modified: accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java URL: http://svn.apache.org/viewvc/accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java (original) +++ accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java Thu Jan 3 19:18:05 2013 @@ -3065,14 +3065,14 @@ public class TabletServer extends Abstra } @Override - public Set getContextURIs(String context) { + public String getContextURIs(String context) { String key = Property.VFS_CONTEXT_CLASSPATH_PROPERTY.getKey() + context; Iterator> iter = getSystemConfiguration().iterator(); while (iter.hasNext()) { Entry entry = iter.next(); if (entry.getKey().equals(key)) { - return new HashSet(Arrays.asList(entry.getValue().split(","))); + return entry.getValue(); } } Modified: accumulo/trunk/start/pom.xml URL: http://svn.apache.org/viewvc/accumulo/trunk/start/pom.xml?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/pom.xml (original) +++ accumulo/trunk/start/pom.xml Thu Jan 3 19:18:05 2013 @@ -115,10 +115,6 @@ commons-logging - org.apache.commons - commons-jci-fam - - commons-io commons-io Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java Thu Jan 3 19:18:05 2013 @@ -19,7 +19,7 @@ package org.apache.accumulo.start; import java.lang.reflect.Method; import java.lang.reflect.Modifier; -import org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader; +import org.apache.accumulo.start.classloader.AccumuloClassLoader; public class Main { @@ -34,39 +34,44 @@ public class Main { final String argsToPass[] = new String[args.length - 1]; System.arraycopy(args, 1, argsToPass, 0, args.length - 1); - Class runTMP = null; + Thread.currentThread().setContextClassLoader(AccumuloClassLoader.getClassLoader()); + + Class vfsClassLoader = AccumuloClassLoader.getClassLoader().loadClass("org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader"); + + ClassLoader cl = (ClassLoader) vfsClassLoader.getMethod("getClassLoader", new Class[] {}).invoke(null, new Object[] {}); - ClassLoader cl = AccumuloVFSClassLoader.getClassLoader(); + Class runTMP = null; + Thread.currentThread().setContextClassLoader(cl); if (args[0].equals("master")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.master.Master"); + runTMP = cl.loadClass("org.apache.accumulo.server.master.Master"); } else if (args[0].equals("tserver")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.tabletserver.TabletServer"); + runTMP = cl.loadClass("org.apache.accumulo.server.tabletserver.TabletServer"); } else if (args[0].equals("shell")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.core.util.shell.Shell"); + runTMP = cl.loadClass("org.apache.accumulo.core.util.shell.Shell"); } else if (args[0].equals("init")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.util.Initialize"); + runTMP = cl.loadClass("org.apache.accumulo.server.util.Initialize"); } else if (args[0].equals("admin")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.util.Admin"); + runTMP = cl.loadClass("org.apache.accumulo.server.util.Admin"); } else if (args[0].equals("gc")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.gc.SimpleGarbageCollector"); + runTMP = cl.loadClass("org.apache.accumulo.server.gc.SimpleGarbageCollector"); } else if (args[0].equals("monitor")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.monitor.Monitor"); + runTMP = cl.loadClass("org.apache.accumulo.server.monitor.Monitor"); } else if (args[0].equals("tracer")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.server.trace.TraceServer"); + runTMP = cl.loadClass("org.apache.accumulo.server.trace.TraceServer"); } else if (args[0].equals("classpath")) { - AccumuloVFSClassLoader.printClassPath(); + vfsClassLoader.getMethod("printClassPath", new Class[] {}).invoke(vfsClassLoader, new Object[] {}); return; } else if (args[0].equals("version")) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.core.Constants"); + runTMP = cl.loadClass("org.apache.accumulo.core.Constants"); System.out.println(runTMP.getField("VERSION").get(null)); return; } else if (args[0].equals("rfile-info") ) { - runTMP = AccumuloVFSClassLoader.loadClass("org.apache.accumulo.core.file.rfile.PrintInfo"); + runTMP = cl.loadClass("org.apache.accumulo.core.file.rfile.PrintInfo"); } else { try { - runTMP = AccumuloVFSClassLoader.loadClass(args[0]); + runTMP = cl.loadClass(args[0]); } catch (ClassNotFoundException cnfe) { System.out.println("Classname " + args[0] + " not found. Please make sure you use the wholly qualified package name."); System.exit(1); @@ -97,8 +102,7 @@ public class Main { }; Thread t = new Thread(r, args[0]); - ClassLoader tcl = AccumuloVFSClassLoader.getClassLoader(); - t.setContextClassLoader(tcl); + t.setContextClassLoader(cl); t.start(); } catch (Throwable t) { System.err.println("Uncaught exception: " + t.getMessage()); Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java Thu Jan 3 19:18:05 2013 @@ -1,4 +1,4 @@ -/* +/** * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. @@ -24,20 +24,14 @@ import java.net.URI; import java.net.URISyntaxException; import java.net.URL; import java.net.URLClassLoader; -import java.security.AccessController; -import java.security.PrivilegedAction; import java.util.ArrayList; -import java.util.HashSet; import java.util.Map; -import java.util.Set; import java.util.regex.Matcher; import java.util.regex.Pattern; import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; -import org.apache.commons.jci.listeners.AbstractFilesystemAlterationListener; -import org.apache.commons.jci.monitor.FilesystemAlterationObserver; import org.apache.log4j.Logger; import org.w3c.dom.Document; import org.w3c.dom.Element; @@ -45,189 +39,129 @@ import org.w3c.dom.Node; import org.w3c.dom.NodeList; /** - * This class builds a hierarchy of Classloaders in the form of: - * - * 
- * SystemClassLoader
- *       ^
- *       |
- * URLClassLoader that references the URLs for HADOOP_HOME, ZOOKEEPER_HOME, ACCUMULO_HOME and their associated directories
- *       ^
- *       |
- * URLClassLoader that references ACCUMULO_HOME/lib/ext, locations from the ACCUMULO property general.dynamic.classpaths and $ACCUMULO_XTRAJARS
- *
- * - * The last URLClassLoader in the chain monitors the directories for changes every 3 seconds (default value). If a change occurs, the last URLClassLoader is - * abandoned and a new one is created. Objects that still reference the abandoned URLClassLoader will cause it to not be garbage collected until those objects - * are garbage collected. If reload happens often enough it may cause excessive memory usage. * */ public class AccumuloClassLoader { - private static class Listener extends AbstractFilesystemAlterationListener { - - private volatile boolean firstCall = true; - - @Override - public void onStop(FilesystemAlterationObserver pObserver) { - - super.onStop(pObserver); - - if (firstCall) { - // the first time this is called it reports everything that exist as created - // so there is no need to do anything - synchronized (this) { - firstCall = false; - this.notifyAll(); - } - return; - } - - if (super.getChangedFiles().size() > 0 || super.getCreatedFiles().size() > 0 || super.getDeletedFiles().size() > 0 - || super.getChangedDirectories().size() > 0 || super.getCreatedDirectories().size() > 0 || super.getDeletedDirectories().size() > 0) { - log.debug("Files have changed, setting loader to null "); - loader = null; - } - - } - - public synchronized void waitForFirstCall() { - while (firstCall == true) { - try { - this.wait(); - } catch (InterruptedException e) {} - } - } - } - - private static final Logger log = Logger.getLogger(AccumuloClassLoader.class); - public static final String CLASSPATH_PROPERTY_NAME = "general.classpaths"; - - public static final String DYNAMIC_CLASSPATH_PROPERTY_NAME = "general.dynamic.classpaths"; - + public static final String ACCUMULO_CLASSPATH_VALUE = "$ACCUMULO_HOME/conf,\n" + "$ACCUMULO_HOME/lib/[^.].$ACCUMULO_VERSION.jar,\n" + "$ACCUMULO_HOME/lib/[^.].*.jar,\n" + "$ZOOKEEPER_HOME/zookeeper[^.].*.jar,\n" + "$HADOOP_HOME/[^.].*.jar,\n" + "$HADOOP_HOME/conf,\n" + "$HADOOP_HOME/lib/[^.].*.jar,\n"; - /** - * Dynamic classpath. These locations will be monitored for changes. - */ - public static final String DEFAULT_DYNAMIC_CLASSPATH_VALUE = "$ACCUMULO_HOME/lib/ext/[^.].*.jar\n"; - - public static final String DEFAULT_CLASSPATH_VALUE = ACCUMULO_CLASSPATH_VALUE; + private static String SITE_CONF; - private static final String SITE_CONF; + private static URLClassLoader classloader; + + private static Logger log = Logger.getLogger(AccumuloClassLoader.class); + static { String configFile = System.getProperty("org.apache.accumulo.config.file", "accumulo-site.xml"); if (System.getenv("ACCUMULO_HOME") != null) { // accumulo home should be set SITE_CONF = System.getenv("ACCUMULO_HOME") + "/conf/" + configFile; } else { - /* - * if not it must be the build-server in which case I use a hack to get unittests working - */ - String userDir = System.getProperty("user.dir"); - if (userDir == null) - throw new RuntimeException("Property user.dir is not set"); - int index = userDir.indexOf("accumulo/"); - if (index >= 0) { - String acuhome = userDir.substring(0, index + "accumulo/".length()); - SITE_CONF = acuhome + "/conf/" + configFile; - } else { - SITE_CONF = "/conf/" + configFile; + try { + URL siteUrl = AccumuloClassLoader.getClassLoader().getResource(configFile); + if (siteUrl != null) + SITE_CONF = siteUrl.toString(); + else + SITE_CONF = null; + } catch (IOException e) { + throw new RuntimeException(e); } } - + + //Register the shutdown hook + // TODO + // Runtime.getRuntime().addShutdownHook(new Thread(new AccumuloVFSClassLoaderShutdownThread())); } - - private static ClassLoader parent = null; - private static volatile ClassLoader loader = null; - private static AccumuloFilesystemAlterationMonitor monitor = null; - private static final Object lock = new Object(); - - private static ArrayList findDynamicURLs() throws IOException { - StringBuilder cp = new StringBuilder(getAccumuloDynamicClasspathStrings()); - String envJars = System.getenv("ACCUMULO_XTRAJARS"); - if (null != envJars && !envJars.equals("")) - cp = cp.append(",").append(envJars); - ArrayList urls = new ArrayList(); - for (String classpath : cp.toString().split(",")) { - if (!classpath.startsWith("#")) { - addUrl(replaceEnvVars(classpath, System.getenv()), urls); + + /** + * Parses and XML Document for a property node for a with the value propertyName if it finds one the function return that property's value for its + * node. If not found the function will return null + * + * @param d + * XMLDocument to search through + * @param propertyName + */ + private static String getAccumuloClassPathStrings(Document d, String propertyName) { + NodeList pnodes = d.getElementsByTagName("property"); + for (int i = pnodes.getLength() - 1; i >= 0; i--) { + Element current_property = (Element) pnodes.item(i); + Node cname = current_property.getElementsByTagName("name").item(0); + if (cname != null && cname.getTextContent().compareTo(propertyName) == 0) { + Node cvalue = current_property.getElementsByTagName("value").item(0); + if (cvalue != null) { + return cvalue.getTextContent(); + } } } - return urls; + return null; } + + /** + * Looks for the site configuration file for Accumulo and if it has a property for propertyName return it otherwise returns defaultValue Should throw an + * exception if the default configuration can not be read; + * + * @param propertyName + * Name of the property to pull + * @param defaultValue + * Value to default to if not found. + * @return site or default class path String + */ - private static Set findDirsFromUrls() throws IOException { - Set dirs = new HashSet(); - StringBuilder cp = new StringBuilder(getAccumuloDynamicClasspathStrings()); - String envJars = System.getenv("ACCUMULO_XTRAJARS"); - if (null != envJars && !envJars.equals("")) - cp = cp.append(",").append(envJars); - ArrayList urls = new ArrayList(); - for (String classpath : cp.toString().split(",")) { - if (!classpath.startsWith("#")) { - classpath = classpath.trim(); - if (classpath.length() == 0) - continue; - - classpath = replaceEnvVars(classpath, System.getenv()); - if (classpath == null) - continue; - // Try to make a URI out of the classpath - URI uri = null; - try { - uri = new URI(classpath); - } catch (URISyntaxException e) { - // Not a valid URI - } - - if (null == uri || !uri.isAbsolute() || (null != uri.getScheme() && uri.getScheme().equals("file://"))) { - // Then treat this URI as a File. - // This checks to see if the url string is a dir if it expand and get all jars in that directory - final File extDir = new File(classpath); - if (extDir.isDirectory()) - urls.add(extDir.toURI().toURL()); - else { - urls.add(extDir.getAbsoluteFile().getParentFile().toURI().toURL()); - } - } else { - urls.add(uri.toURL()); - } - } - } + public static String getAccumuloString(String propertyName, String defaultValue) { - for (URL url : urls) { + try { + DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); + DocumentBuilder db = dbf.newDocumentBuilder(); + String site_classpath_string = null; try { - File f = new File(url.toURI()); - if (!f.isDirectory()) - f = f.getParentFile(); - dirs.add(f); - } catch (URISyntaxException e) { - log.error("Unable to find directory for " + url + ", cannot create URI from it"); + Document site_conf = db.parse(SITE_CONF); + site_classpath_string = getAccumuloClassPathStrings(site_conf, propertyName); + } catch (Exception e) { + /* we don't care because this is optional and we can use defaults */ } + if (site_classpath_string != null) + return site_classpath_string; + return defaultValue; + } catch (Exception e) { + throw new IllegalStateException("ClassPath Strings Lookup failed", e); } - return dirs; } - private static ArrayList findAccumuloURLs() throws IOException { - String cp = getAccumuloClasspathStrings(); - if (cp == null) - return new ArrayList(); - ArrayList urls = new ArrayList(); - for (String classpath : cp.split(",")) { - if (!classpath.startsWith("#")) { - addUrl(replaceEnvVars(classpath, System.getenv()), urls); + /** + * Replace environment variables in the classpath string with their actual value + * + * @param classpath + * @param env + * @return + */ + public static String replaceEnvVars(String classpath, Map env) { + Pattern envPat = Pattern.compile("\\$[A-Za-z][a-zA-Z0-9_]*"); + Matcher envMatcher = envPat.matcher(classpath); + while (envMatcher.find(0)) { + // name comes after the '$' + String varName = envMatcher.group().substring(1); + String varValue = env.get(varName); + if (varValue == null) { + varValue = ""; } + classpath = (classpath.substring(0, envMatcher.start()) + varValue + classpath.substring(envMatcher.end())); + envMatcher.reset(classpath); } - return urls; + return classpath; } - + + /** + * Populate the list of URLs with the items in the classpath string + * + * @param classpath + * @param urls + * @throws MalformedURLException + */ private static void addUrl(String classpath, ArrayList urls) throws MalformedURLException { - if (classpath == null) - return; classpath = classpath.trim(); if (classpath.length() == 0) return; @@ -266,171 +200,48 @@ public class AccumuloClassLoader { } } - - private static String replaceEnvVars(String classpath, Map env) { - Pattern envPat = Pattern.compile("\\$[A-Za-z][a-zA-Z0-9_]*"); - Matcher envMatcher = envPat.matcher(classpath); - while (envMatcher.find(0)) { - // name comes after the '$' - String varName = envMatcher.group().substring(1); - String varValue = env.get(varName); - if (varValue == null) { - return null; - } - classpath = (classpath.substring(0, envMatcher.start()) + varValue + classpath.substring(envMatcher.end())); - envMatcher.reset(classpath); - } - return classpath; - } - - public static String getAccumuloDynamicClasspathStrings() throws IllegalStateException { - return getAccumuloString(DYNAMIC_CLASSPATH_PROPERTY_NAME, DEFAULT_DYNAMIC_CLASSPATH_VALUE); - } - - public static String getAccumuloClasspathStrings() throws IllegalStateException { - return getAccumuloString(CLASSPATH_PROPERTY_NAME, ACCUMULO_CLASSPATH_VALUE); - } - - /** - * Looks for the site configuration file for Accumulo and if it has a property for propertyName return it otherwise returns defaultValue Should throw an - * exception if the default configuration can not be read; - * - * @param propertyName - * Name of the property to pull - * @param defaultValue - * Value to default to if not found. - * @return site or default class path String - */ - - private static String getAccumuloString(String propertyName, String defaultValue) { - try { - DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); - DocumentBuilder db = dbf.newDocumentBuilder(); - String site_classpath_string = null; - try { - Document site_conf = db.parse(SITE_CONF); - site_classpath_string = getAccumuloClassPathStrings(site_conf, propertyName); - } catch (Exception e) { - /* we don't care because this is optional and we can use defaults */ - } - if (site_classpath_string != null) - return site_classpath_string; - return defaultValue; - } catch (Exception e) { - throw new IllegalStateException("ClassPath Strings Lookup failed", e); - } - } - - /** - * Parses and XML Document for a property node for a with the value propertyName if it finds one the function return that property's value for its - * node. If not found the function will return null - * - * @param d - * XMLDocument to search through - * @param propertyName - */ - private static String getAccumuloClassPathStrings(Document d, String propertyName) { - NodeList pnodes = d.getElementsByTagName("property"); - for (int i = pnodes.getLength() - 1; i >= 0; i--) { - Element current_property = (Element) pnodes.item(i); - Node cname = current_property.getElementsByTagName("name").item(0); - if (cname != null && cname.getTextContent().compareTo(propertyName) == 0) { - Node cvalue = current_property.getElementsByTagName("value").item(0); - if (cvalue != null) { - return cvalue.getTextContent(); - } - } - } - return null; - } - - public static void printClassPath() { - try { - System.out.println("Accumulo List of classpath items are:"); - for (URL url : findDynamicURLs()) { - System.out.println(url.toExternalForm()); - } - for (URL url : findAccumuloURLs()) { - System.out.println(url.toExternalForm()); + + private static ArrayList findAccumuloURLs() throws IOException { + String cp = getAccumuloString(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.ACCUMULO_CLASSPATH_VALUE); + if (cp == null) + return new ArrayList(); + String[] cps = replaceEnvVars(cp, System.getenv()).split(","); + ArrayList urls = new ArrayList(); + for (String classpath : cps) { + if (!classpath.startsWith("#")) { + addUrl(classpath, urls); } - } catch (Throwable t) { - throw new RuntimeException(t); } + return urls; } - public synchronized static Class loadClass(String classname, Class extension) throws ClassNotFoundException { - try { - return (Class) Class.forName(classname, true, getClassLoader()).asSubclass(extension); - } catch (IOException e) { - throw new ClassNotFoundException("IO Error loading class " + classname, e); - } - } - - public static Class loadClass(String classname) throws ClassNotFoundException { - return loadClass(classname, Object.class).asSubclass(Object.class); - } - - private static ClassLoader getAccumuloClassLoader() throws IOException { - ClassLoader parentClassLoader = ClassLoader.getSystemClassLoader(); - ArrayList accumuloURLs = findAccumuloURLs(); - log.debug("Create Dependency ClassLoader using URLs: " + accumuloURLs.toString()); - URLClassLoader aClassLoader = new URLClassLoader(accumuloURLs.toArray(new URL[accumuloURLs.size()]), parentClassLoader); - return aClassLoader; - } - - public static ClassLoader getClassLoader() throws IOException { - ClassLoader localLoader = loader; - while (null == localLoader) { - synchronized (lock) { - if (null == loader) { - - if (null != monitor) { - monitor.stop(); - monitor = null; - } - - if (null == parent) - parent = getAccumuloClassLoader(); - - // Find the dynamic classpath items - final ArrayList dynamicURLs = findDynamicURLs(); - // Find the directories for the dynamic classpath items - Set monitoredDirs = findDirsFromUrls(); - - // Setup the directory monitor - monitor = new AccumuloFilesystemAlterationMonitor(); - Listener myListener = new Listener(); - for (File dir : monitoredDirs) { - if (monitor.getListenersFor(dir) != null || monitor.getListenersFor(dir).length > 0) { - log.debug("Monitor listening to " + dir.getAbsolutePath()); - monitor.addListener(dir, myListener); - } - } - // setting this interval below 1 second is probably not helpful because the file modification time is in seconds - monitor.setInterval(1000); - monitor.start(); - if (monitoredDirs.size() != 0) - myListener.waitForFirstCall(); - log.debug("Create Dynamic ClassLoader using URLs: " + dynamicURLs.toString()); - // the file system listner must run once to get the state of the filesystem, and it only detects changes on subsequent runs - // we need to check to see if the set of files has changed since we started monitoring, so that we know we are monitoring all of the files - // in the - // class loader - HashSet checkDynamicURLs = new HashSet(findDynamicURLs()); - HashSet originalDynamicURLs = new HashSet(dynamicURLs); - if (checkDynamicURLs.equals(originalDynamicURLs)) { - // file set has not changed, so we know that all of the dynamicURLs are being monitored for changes - loader = AccessController.doPrivileged(new PrivilegedAction() { - public ClassLoader run() { - return new URLClassLoader(dynamicURLs.toArray(new URL[dynamicURLs.size()]), parent); + public static synchronized ClassLoader getClassLoader() throws IOException { + if (classloader == null) { + ArrayList urls = findAccumuloURLs(); + + ClassLoader parentClassLoader = ClassLoader.getSystemClassLoader(); + + log.debug("Create 2nd tier ClassLoader using URLs: " + urls.toString()); + URLClassLoader aClassLoader = new URLClassLoader(urls.toArray(new URL[urls.size()]), parentClassLoader) { + protected synchronized Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + + if (name.startsWith("org.apache.accumulo.start.classloader.vfs")) { + Class c = findLoadedClass(name); + if (c == null) { + try { + // try finding this class here instead of parent + c = findClass(name); + } catch (ClassNotFoundException e) { + } - }); + } } + return super.loadClass(name, resolve); } - } - localLoader = loader; + }; + classloader = aClassLoader; } - return localLoader; + + return classloader; } - } Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java Thu Jan 3 19:18:05 2013 @@ -20,6 +20,7 @@ import java.io.IOException; import java.io.InputStream; import java.net.URL; import java.security.SecureClassLoader; +import java.util.ArrayList; import java.util.Enumeration; import org.apache.commons.vfs2.FileChangeEvent; @@ -44,21 +45,46 @@ public class AccumuloReloadingVFSClassLo /* 5 minute timeout */ private static final int DEFAULT_TIMEOUT = 300000; - private FileObject[] files = null; + private String uris; + private FileObject[] files; private FileSystemManager vfs = null; private ClassLoader parent = null; private DefaultFileMonitor monitor = null; private volatile VFSClassLoader cl = null; - - public AccumuloReloadingVFSClassLoader(FileObject[] files, FileSystemManager vfs, ClassLoader parent) throws FileSystemException { - this(files, vfs, parent, DEFAULT_TIMEOUT); + + private synchronized VFSClassLoader getClassloader() { + if (cl == null) { + try { + files = AccumuloVFSClassLoader.resolve(vfs, uris); + + if (null != parent) + setClassloader(new VFSClassLoader(files, vfs, parent)); + else + setClassloader(new VFSClassLoader(files, vfs)); + + } catch (FileSystemException fse) { + throw new RuntimeException(fse); + } + } + + return cl; } - public AccumuloReloadingVFSClassLoader(FileObject[] files, FileSystemManager vfs, ClassLoader parent, long monitorDelay) throws FileSystemException { - this.files = files; + private synchronized void setClassloader(VFSClassLoader cl) { + this.cl = cl; + + } + + public AccumuloReloadingVFSClassLoader(String uris, FileSystemManager vfs, ClassLoader parent, long monitorDelay) throws FileSystemException { + super(parent); + + this.uris = uris; this.vfs = vfs; this.parent = parent; + ArrayList pathsToMonitor = new ArrayList(); + files = AccumuloVFSClassLoader.resolve(vfs, uris, pathsToMonitor); + if (null != parent) cl = new VFSClassLoader(files, vfs, parent); else @@ -66,12 +92,18 @@ public class AccumuloReloadingVFSClassLo monitor = new DefaultFileMonitor(this); monitor.setDelay(monitorDelay); - monitor.setRecursive(true); - for (FileObject file : files) + monitor.setRecursive(false); + for (FileObject file : pathsToMonitor) { monitor.addFile(file); + log.debug("monitoring " + file); + } monitor.start(); } + public AccumuloReloadingVFSClassLoader(String uris, FileSystemManager vfs, ClassLoader parent) throws FileSystemException { + this(uris, vfs, parent, DEFAULT_TIMEOUT); + } + public FileObject[] getFiles() { return this.files; } @@ -86,24 +118,24 @@ public class AccumuloReloadingVFSClassLo public void fileCreated(FileChangeEvent event) throws Exception { if (log.isDebugEnabled()) log.debug(event.getFile().getURL().toString() + " created, recreating classloader"); - cl = new VFSClassLoader(files, vfs, parent); + setClassloader(null); } public void fileDeleted(FileChangeEvent event) throws Exception { if (log.isDebugEnabled()) log.debug(event.getFile().getURL().toString() + " changed, recreating classloader"); - cl = new VFSClassLoader(files, vfs, parent); + setClassloader(null); } public void fileChanged(FileChangeEvent event) throws Exception { if (log.isDebugEnabled()) log.debug(event.getFile().getURL().toString() + " deleted, recreating classloader"); - cl = new VFSClassLoader(files, vfs, parent); + setClassloader(null); } @Override public Class loadClass(String name) throws ClassNotFoundException { - return this.cl.loadClass(name); + return this.getClassloader().loadClass(name); } @Override @@ -118,37 +150,37 @@ public class AccumuloReloadingVFSClassLo @Override public URL getResource(String name) { - return this.cl.getResource(name); + return this.getClassloader().getResource(name); } @Override public Enumeration getResources(String name) throws IOException { - return this.cl.getResources(name); + return this.getClassloader().getResources(name); } @Override public InputStream getResourceAsStream(String name) { - return this.cl.getResourceAsStream(name); + return this.getClassloader().getResourceAsStream(name); } @Override public synchronized void setDefaultAssertionStatus(boolean enabled) { - this.cl.setDefaultAssertionStatus(enabled); + this.getClassloader().setDefaultAssertionStatus(enabled); } @Override public synchronized void setPackageAssertionStatus(String packageName, boolean enabled) { - this.cl.setPackageAssertionStatus(packageName, enabled); + this.getClassloader().setPackageAssertionStatus(packageName, enabled); } @Override public synchronized void setClassAssertionStatus(String className, boolean enabled) { - this.cl.setClassAssertionStatus(className, enabled); + this.getClassloader().setClassAssertionStatus(className, enabled); } @Override public synchronized void clearAssertionStatus() { - this.cl.clearAssertionStatus(); + this.getClassloader().clearAssertionStatus(); } @Override @@ -165,5 +197,7 @@ public class AccumuloReloadingVFSClassLo return buf.toString(); } + + } Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java Thu Jan 3 19:18:05 2013 @@ -16,30 +16,25 @@ */ package org.apache.accumulo.start.classloader.vfs; -import java.io.File; -import java.io.FilenameFilter; import java.io.IOException; -import java.net.MalformedURLException; -import java.net.URI; -import java.net.URISyntaxException; import java.net.URL; import java.net.URLClassLoader; import java.util.ArrayList; -import java.util.Map; -import java.util.regex.Matcher; -import java.util.regex.Pattern; +import java.util.Collections; import org.apache.accumulo.start.classloader.AccumuloClassLoader; +import org.apache.accumulo.start.classloader.vfs.providers.ReadOnlyHdfsFileProvider; import org.apache.commons.vfs2.CacheStrategy; import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; import org.apache.commons.vfs2.FileType; import org.apache.commons.vfs2.cache.DefaultFilesCache; import org.apache.commons.vfs2.cache.SoftRefFilesCache; import org.apache.commons.vfs2.impl.DefaultFileReplicator; import org.apache.commons.vfs2.impl.DefaultFileSystemManager; import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory; -import org.apache.commons.vfs2.provider.ReadOnlyHdfsFileProvider; -import org.apache.hadoop.conf.Configuration; +import org.apache.commons.vfs2.impl.VFSClassLoader; import org.apache.log4j.Logger; /** @@ -49,17 +44,16 @@ import org.apache.log4j.Logger; * SystemClassLoader that loads JVM classes * ^ * | - * URLClassLoader that references the URLs for HADOOP_HOME, ZOOKEEPER_HOME, ACCUMULO_HOME and their associated directories + * AccumuloClassLoader loads jars from locations in general.classpaths. Usually the URLs for HADOOP_HOME, ZOOKEEPER_HOME, ACCUMULO_HOME and their associated directories * ^ * | - * AccumuloContextClassLoader that contains a map of context names to AccumuloReloadingVFSClassLoaders - * + * VFSClassLoader that loads jars from locations in general.vfs.classpaths. Can be used to load accumulo jar from HDFS + * ^ + * | + * AccumuloReloadingVFSClassLoader That loads jars from locations in general.dynamic.classpaths. Used to load jar dynamically. * - * This class requires new properties in the site configuration file + * * - * default.context.classpath -> list of URIs for the default system context. - * classloader.context.names -> name1, name2, name3 - * .context.classpath -> list of URIs for this context * */ public class AccumuloVFSClassLoader { @@ -71,157 +65,110 @@ public class AccumuloVFSClassLoader { } } - - private static final Logger log = Logger.getLogger(AccumuloVFSClassLoader.class); - public static final String VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY = "classloader.vfs.context.classpath.system"; + public static final String DYNAMIC_CLASSPATH_PROPERTY_NAME = "general.dynamic.classpaths"; - public static final String VFS_CLASSLOADER_CONTEXT_NAMES_PROPERTY = "classloader.vfs.context.names"; + public static final String DEFAULT_DYNAMIC_CLASSPATH_VALUE = "$ACCUMULO_HOME/lib/ext/[^.].*.jar\n"; + + public static final String VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY = "general.vfs.classpaths"; - public static final String VFS_CONTEXT_CLASSPATH_PROPERTY = "classloader.vfs.context.classpath."; + public static final String VFS_CONTEXT_CLASSPATH_PROPERTY = "general.vfs.context.classpath."; private static DefaultFileSystemManager vfs = null; private static ClassLoader parent = null; - private static volatile AccumuloReloadingVFSClassLoader loader = null; + private static volatile ClassLoader loader = null; private static final Object lock = new Object(); - private static final Configuration ACC_CONF = new Configuration(); - private static final String SITE_CONF; + private static ContextManager contextManager; - static { - String configFile = System.getProperty("org.apache.accumulo.config.file", "accumulo-site.xml"); - if (System.getenv("ACCUMULO_HOME") != null) { - // accumulo home should be set - SITE_CONF = System.getenv("ACCUMULO_HOME") + "/conf/" + configFile; - } else { - /* - * if not it must be the build-server in which case I use a hack to get unittests working - */ - String userDir = System.getProperty("user.dir"); - if (userDir == null) - throw new RuntimeException("Property user.dir is not set"); - int index = userDir.indexOf("accumulo/"); - if (index >= 0) { - String acuhome = userDir.substring(0, index + "accumulo/".length()); - SITE_CONF = acuhome + "/conf/" + configFile; - } else { - SITE_CONF = "/conf/" + configFile; - } - } + private static Logger log = Logger.getLogger(AccumuloVFSClassLoader.class); + + public synchronized static Class loadClass(String classname, Class extension) throws ClassNotFoundException { try { - File siteFile = new File(SITE_CONF); - if (siteFile.exists()) - ACC_CONF.addResource(siteFile.toURI().toURL()); - } catch (MalformedURLException e) { - throw new RuntimeException("Unable to create configuration from accumulo-site file: " + SITE_CONF, e); + return (Class) getClassLoader().loadClass(classname).asSubclass(extension); + } catch (IOException e) { + throw new ClassNotFoundException("IO Error loading class " + classname, e); } - - //Register the shutdown hook - Runtime.getRuntime().addShutdownHook(new Thread(new AccumuloVFSClassLoaderShutdownThread())); + } + + public static Class loadClass(String classname) throws ClassNotFoundException { + return loadClass(classname, Object.class).asSubclass(Object.class); } - /** - * Replace environment variables in the classpath string with their actual value - * - * @param classpath - * @param env - * @return - */ - private static String replaceEnvVars(String classpath, Map env) { - Pattern envPat = Pattern.compile("\\$[A-Za-z][a-zA-Z0-9_]*"); - Matcher envMatcher = envPat.matcher(classpath); - while (envMatcher.find(0)) { - // name comes after the '$' - String varName = envMatcher.group().substring(1); - String varValue = env.get(varName); - if (varValue == null) { - varValue = ""; - } - classpath = (classpath.substring(0, envMatcher.start()) + varValue + classpath.substring(envMatcher.end())); - envMatcher.reset(classpath); - } - return classpath; + static FileObject[] resolve(FileSystemManager vfs, String uris) throws FileSystemException { + return resolve(vfs, uris, new ArrayList()); } - /** - * Populate the list of URLs with the items in the classpath string - * - * @param classpath - * @param urls - * @throws MalformedURLException - */ - private static void addUrl(String classpath, ArrayList urls) throws MalformedURLException { - classpath = classpath.trim(); - if (classpath.length() == 0) - return; - - classpath = replaceEnvVars(classpath, System.getenv()); - - // Try to make a URI out of the classpath - URI uri = null; - try { - uri = new URI(classpath); - } catch (URISyntaxException e) { - // Not a valid URI - } - - if (null == uri || !uri.isAbsolute() || (null != uri.getScheme() && uri.getScheme().equals("file://"))) { - // Then treat this URI as a File. - // This checks to see if the url string is a dir if it expand and get all jars in that directory - final File extDir = new File(classpath); - if (extDir.isDirectory()) - urls.add(extDir.toURI().toURL()); - else { - if (extDir.getParentFile() != null) { - File[] extJars = extDir.getParentFile().listFiles(new FilenameFilter() { - public boolean accept(File dir, String name) { - return name.matches("^" + extDir.getName()); + static FileObject[] resolve(FileSystemManager vfs, String uris, ArrayList pathsToMonitor) throws FileSystemException { + if (uris == null) + return new FileObject[0]; + + ArrayList classpath = new ArrayList(); + + pathsToMonitor.clear(); + + for (String path : uris.split(",")) { + + path = path.trim(); + + if (path.equals("")) + continue; + + path = AccumuloClassLoader.replaceEnvVars(path, System.getenv()); + + FileObject fo = vfs.resolveFile(path); + + switch (fo.getType()) { + case FILE: + classpath.add(fo); + pathsToMonitor.add(fo); + break; + case FOLDER: + pathsToMonitor.add(fo); + for (FileObject child : fo.getChildren()) { + classpath.add(child); + } + break; + case IMAGINARY: + // assume its a pattern + String pattern = fo.getName().getBaseName(); + if (fo.getParent() != null && fo.getParent().getType() == FileType.FOLDER) { + pathsToMonitor.add(fo.getParent()); + FileObject[] children = fo.getParent().getChildren(); + for (FileObject child : children) { + if (child.getType() == FileType.FILE && child.getName().getBaseName().matches(pattern)) { + classpath.add(child); + } } - }); - if (extJars != null && extJars.length > 0) { - for (File jar : extJars) - urls.add(jar.toURI().toURL()); + } else { + log.warn("ignoring classpath entry " + fo); } - } + break; + default: + log.warn("ignoring classpath entry " + fo); + break; } - } else { - urls.add(uri.toURL()); + } - + + return classpath.toArray(new FileObject[classpath.size()]); } + + private static ClassLoader createDynamicClassloader(ClassLoader parent) throws FileSystemException, IOException { + String dynamicCPath = AccumuloClassLoader.getAccumuloString(DYNAMIC_CLASSPATH_PROPERTY_NAME, DEFAULT_DYNAMIC_CLASSPATH_VALUE); - private static ArrayList findAccumuloURLs() throws IOException { - String cp = ACC_CONF.get(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.ACCUMULO_CLASSPATH_VALUE); - if (cp == null) - return new ArrayList(); - String[] cps = replaceEnvVars(cp, System.getenv()).split(","); - ArrayList urls = new ArrayList(); - for (String classpath : cps) { - if (!classpath.startsWith("#")) { - addUrl(classpath, urls); - } - } - return urls; - } - - private static ClassLoader getAccumuloClassLoader() throws IOException { - ClassLoader parentClassLoader = ClassLoader.getSystemClassLoader(); - ArrayList accumuloURLs = findAccumuloURLs(); - log.debug("Create 2nd tier ClassLoader using URLs: " + accumuloURLs.toString()); - URLClassLoader aClassLoader = new URLClassLoader(accumuloURLs.toArray(new URL[accumuloURLs.size()]), parentClassLoader); - return aClassLoader; - } + String envJars = System.getenv("ACCUMULO_XTRAJARS"); + if (null != envJars && !envJars.equals("")) + if (dynamicCPath != null && !dynamicCPath.equals("")) + dynamicCPath = dynamicCPath + "," + envJars; + else + dynamicCPath = envJars; + + if (dynamicCPath == null || dynamicCPath.equals("")) + return parent; - public synchronized static Class loadClass(String classname, Class extension) throws ClassNotFoundException { - try { - return (Class) getClassLoader().loadClass(classname).asSubclass(extension); - } catch (IOException e) { - throw new ClassNotFoundException("IO Error loading class " + classname, e); - } - } - - public static Class loadClass(String classname) throws ClassNotFoundException { - return loadClass(classname, Object.class).asSubclass(Object.class); + // TODO monitor time for lib/ext was 1 sec... should this be configurable? + return new AccumuloReloadingVFSClassLoader(dynamicCPath, vfs, parent, 1000); } public static ClassLoader getClassLoader() throws IOException { @@ -273,33 +220,24 @@ public class AccumuloVFSClassLoader { } // Set up the 2nd tier class loader - if (null == parent) - parent = getAccumuloClassLoader(); + if (null == parent) { + parent = AccumuloClassLoader.getClassLoader(); + } - // Get the default context classpaths from the configuration - String[] defaultPaths = ACC_CONF.getStrings(VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY); - if (null == defaultPaths || defaultPaths.length == 0) { - localLoader = parent; + FileObject[] vfsCP = resolve(vfs, AccumuloClassLoader.getAccumuloString(VFS_CLASSLOADER_SYSTEM_CLASSPATH_PROPERTY, "")); + + if (vfsCP.length == 0) { + localLoader = createDynamicClassloader(parent); + loader = localLoader; return localLoader; } - - ArrayList defaultClassPath = new ArrayList(); - for (String path : defaultPaths) { - FileObject fo = vfs.resolveFile(path); - if (fo.getType().equals(FileType.FILE)) { - defaultClassPath.add(fo); - } else { - for (FileObject child : fo.getChildren()) { - defaultClassPath.add(child); - } - } - } //Create the Accumulo Context ClassLoader using the DEFAULT_CONTEXT - loader = new AccumuloReloadingVFSClassLoader(defaultClassPath.toArray(new FileObject[defaultClassPath.size()]), vfs, parent); + localLoader = new VFSClassLoader(vfsCP, vfs, parent); + localLoader = createDynamicClassloader(localLoader); + loader = localLoader; } } - localLoader = loader; } return localLoader; } @@ -307,24 +245,45 @@ public class AccumuloVFSClassLoader { public static void printClassPath() { try { ClassLoader cl = getClassLoader(); - if (cl instanceof URLClassLoader) { - //If VFS class loader enabled, but no contexts defined. - URLClassLoader ucl = (URLClassLoader) cl; - System.out.println("URL classpath items are: \n"); - for (URL u : ucl.getURLs()) { - System.out.println("\t" + u.toExternalForm()); - } - } else if (cl instanceof AccumuloReloadingVFSClassLoader) { - //If VFS class loader enabled and contexts are defined - System.out.println("URL classpath items are: \n"); - URLClassLoader ucl = (URLClassLoader) cl.getParent(); - for (URL u : ucl.getURLs()) { - System.out.println("\t" + u.toExternalForm()); + ArrayList classloaders = new ArrayList(); + + while (cl != null) { + classloaders.add(cl); + cl = cl.getParent(); + } + + Collections.reverse(classloaders); + + int level = 0; + + for (ClassLoader classLoader : classloaders) { + if (level > 0) + System.out.println(); + System.out.print("Level " + level + " "); + level++; + + if (classLoader instanceof URLClassLoader) { + // If VFS class loader enabled, but no contexts defined. + URLClassLoader ucl = (URLClassLoader) classLoader; + System.out.println("URL classpath items are:"); + + for (URL u : ucl.getURLs()) { + System.out.println("\t" + u.toExternalForm()); + } + + } else if (classLoader instanceof AccumuloReloadingVFSClassLoader) { + System.out.println("VFS classpaths items are:\n" + classLoader.toString()); + } else if (classLoader instanceof VFSClassLoader) { + System.out.println("VFS classpaths items are:"); + VFSClassLoader vcl = (VFSClassLoader) classLoader; + for (FileObject f : vcl.getFileObjects()) { + System.out.println("\t" + f.getURL().toExternalForm()); + } + } else { + System.out.println("Unknown classloader configuration " + classLoader.getClass()); } - System.out.println("VFS classpaths items are:\n" + getClassLoader().toString()); - } else { - System.out.println("Unknown classloader configuration"); } + } catch (Throwable t) { throw new RuntimeException(t); } Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java Thu Jan 3 19:18:05 2013 @@ -17,47 +17,31 @@ package org.apache.accumulo.start.classloader.vfs; import java.io.IOException; -import java.util.ArrayList; import java.util.HashMap; -import java.util.HashSet; import java.util.Map; import java.util.Set; -import org.apache.commons.vfs2.FileObject; import org.apache.commons.vfs2.FileSystemException; import org.apache.commons.vfs2.FileSystemManager; -import org.apache.commons.vfs2.FileType; public class ContextManager { // there is a lock per context so that one context can initialize w/o blocking another context private class Context { AccumuloReloadingVFSClassLoader loader; - Set uris; + String uris; boolean closed = false; - Context(Set uris) { - this.uris = new HashSet(uris); + Context(String uris) { + this.uris = uris; } synchronized AccumuloReloadingVFSClassLoader getClassLoader() throws FileSystemException { if (closed) return null; - + if (loader == null) { - ArrayList defaultClassPath = new ArrayList(); - for (String path : uris) { - FileObject fo = vfs.resolveFile(path); - if (fo.getType().equals(FileType.FILE)) { - defaultClassPath.add(fo); - } else { - for (FileObject child : fo.getChildren()) { - defaultClassPath.add(child); - } - } - } - - loader = new AccumuloReloadingVFSClassLoader(defaultClassPath.toArray(new FileObject[0]), vfs, parent); + loader = new AccumuloReloadingVFSClassLoader(uris, vfs, parent); } return loader; @@ -82,7 +66,7 @@ public class ContextManager { } public interface ContextConfig { - Set getContextURIs(String context); + String getContextURIs(String context); boolean isIsolated(String context); } @@ -100,7 +84,7 @@ public class ContextManager { public ClassLoader getClassLoader(String contextName) throws FileSystemException { - Set uris = config.getContextURIs(contextName); + String uris = config.getContextURIs(contextName); if (uris == null) throw new IllegalArgumentException("Unknown context " + contextName); Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java?rev=1428556&r1=1427869&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import java.io.FileNotFoundException; import java.io.InputStream; @@ -25,6 +25,8 @@ import org.apache.commons.vfs2.FileObjec import org.apache.commons.vfs2.FileSystemException; import org.apache.commons.vfs2.FileType; import org.apache.commons.vfs2.RandomAccessContent; +import org.apache.commons.vfs2.provider.AbstractFileName; +import org.apache.commons.vfs2.provider.AbstractFileObject; import org.apache.commons.vfs2.util.RandomAccessMode; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java?rev=1428556&r1=1427869&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import org.apache.commons.vfs2.FileSystem; import org.apache.commons.vfs2.FileSystemConfigBuilder; Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java?rev=1428556&r1=1427869&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import org.apache.commons.vfs2.FileContent; import org.apache.commons.vfs2.FileContentInfo; Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java?rev=1428556&r1=1427869&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import java.io.IOException; import java.io.InputStream; Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java?rev=1428556&r1=1427869&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import java.util.Arrays; import java.util.Collection; @@ -26,6 +26,7 @@ import org.apache.commons.vfs2.FileSyste import org.apache.commons.vfs2.FileSystemConfigBuilder; import org.apache.commons.vfs2.FileSystemException; import org.apache.commons.vfs2.FileSystemOptions; +import org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider; import org.apache.commons.vfs2.provider.http.HttpFileNameParser; public class ReadOnlyHdfsFileProvider extends AbstractOriginatingFileProvider { Modified: accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java?rev=1428556&r1=1427869&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java (original) +++ accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import java.io.IOException; import java.util.Collection; @@ -24,6 +24,8 @@ import org.apache.commons.vfs2.FileName; import org.apache.commons.vfs2.FileObject; import org.apache.commons.vfs2.FileSystemException; import org.apache.commons.vfs2.FileSystemOptions; +import org.apache.commons.vfs2.provider.AbstractFileName; +import org.apache.commons.vfs2.provider.AbstractFileSystem; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; Modified: accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java (original) +++ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java Thu Jan 3 19:18:05 2013 @@ -55,7 +55,8 @@ public class AccumuloReloadingVFSClassLo public void testConstructor() throws Exception { FileObject testDir = vfs.resolveFile(TEST_DIR.toUri().toString()); FileObject[] dirContents = testDir.getChildren(); - cl = new AccumuloReloadingVFSClassLoader(dirContents, vfs, ClassLoader.getSystemClassLoader()); + + cl = new AccumuloReloadingVFSClassLoader(TEST_DIR.toUri().toString(), vfs, ClassLoader.getSystemClassLoader()); FileObject[] files = cl.getFiles(); Assert.assertArrayEquals(dirContents, files); } @@ -64,7 +65,8 @@ public class AccumuloReloadingVFSClassLo public void testReloading() throws Exception { FileObject testDir = vfs.resolveFile(TEST_DIR.toUri().toString()); FileObject[] dirContents = testDir.getChildren(); - cl = new AccumuloReloadingVFSClassLoader(dirContents, vfs, ClassLoader.getSystemClassLoader(), 1000); + + cl = new AccumuloReloadingVFSClassLoader(TEST_DIR.toUri().toString(), vfs, ClassLoader.getSystemClassLoader(), 1000); FileObject[] files = cl.getFiles(); Assert.assertArrayEquals(dirContents, files); Modified: accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java (original) +++ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java Thu Jan 3 19:18:05 2013 @@ -20,11 +20,11 @@ import java.io.File; import java.net.URL; import java.net.URLClassLoader; +import org.apache.accumulo.start.classloader.AccumuloClassLoader; import org.apache.accumulo.test.AccumuloDFSBase; -import org.apache.hadoop.conf.Configuration; +import org.apache.commons.vfs2.impl.VFSClassLoader; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.log4j.Logger; import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; @@ -41,33 +41,23 @@ import org.powermock.reflect.Whitebox; "org.apache.hadoop.*"}) public class AccumuloVFSClassLoaderTest extends AccumuloDFSBase { - /* - * Test that if not enabled, the AccumuloClassLoader is used - */ - @Test - public void testNoConfigChanges() throws Exception { - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "ACC_CONF", new Configuration()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "lock", new Object()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "log", Logger.getLogger(AccumuloVFSClassLoader.class)); - ClassLoader acl = AccumuloVFSClassLoader.getClassLoader(); - Assert.assertTrue((acl instanceof URLClassLoader)); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "loader", (AccumuloReloadingVFSClassLoader) null); - } /* * Test that if enabled, but not configured, that the code creates the 2nd level classloader */ @Test public void testDefaultConfig() throws Exception { - Configuration conf = new Configuration(); + + Whitebox.setInternalState(AccumuloVFSClassLoader.class, "loader", (AccumuloReloadingVFSClassLoader) null); + URL defaultDir = this.getClass().getResource("/disabled"); - conf.addResource(new File(defaultDir.getPath() + "/conf/accumulo-site.xml").toURI().toURL()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "ACC_CONF", conf); + + Whitebox.setInternalState(AccumuloClassLoader.class, "SITE_CONF", new File(defaultDir.getPath() + "/conf/accumulo-site.xml").toURI().toURL().toString()); Whitebox.setInternalState(AccumuloVFSClassLoader.class, "lock", new Object()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "log", Logger.getLogger(AccumuloVFSClassLoader.class)); ClassLoader acl = AccumuloVFSClassLoader.getClassLoader(); - Assert.assertTrue((acl instanceof URLClassLoader)); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "loader", (AccumuloReloadingVFSClassLoader) null); + Assert.assertTrue((acl instanceof AccumuloReloadingVFSClassLoader)); + Assert.assertTrue((acl.getParent() instanceof URLClassLoader)); + // URLClassLoader ucl = (URLClassLoader) acl; // URL[] classpath = ucl.getURLs(); // System.out.println(Arrays.toString(classpath)); @@ -79,6 +69,9 @@ public class AccumuloVFSClassLoaderTest @Test public void testDefaultContextConfigured() throws Exception { + + Whitebox.setInternalState(AccumuloVFSClassLoader.class, "loader", (AccumuloReloadingVFSClassLoader) null); + // Create default context directory FileSystem hdfs = cluster.getFileSystem(); Path DEFAULT = new Path("/accumulo/classpath"); @@ -91,17 +84,16 @@ public class AccumuloVFSClassLoaderTest hdfs.copyFromLocalFile(src, dst); URL defaultDir = this.getClass().getResource("/default"); - Configuration conf = new Configuration(hdfs.getConf()); - conf.addResource(new File(defaultDir.getPath() + "/conf/accumulo-site.xml").toURI().toURL()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "ACC_CONF", conf); + + Whitebox.setInternalState(AccumuloClassLoader.class, "SITE_CONF", new File(defaultDir.getPath() + "/conf/accumulo-site.xml").toURI().toURL().toString()); Whitebox.setInternalState(AccumuloVFSClassLoader.class, "lock", new Object()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "log", Logger.getLogger(AccumuloVFSClassLoader.class)); ClassLoader acl = AccumuloVFSClassLoader.getClassLoader(); Assert.assertTrue((acl instanceof AccumuloReloadingVFSClassLoader)); - AccumuloReloadingVFSClassLoader arvcl = (AccumuloReloadingVFSClassLoader) acl; - Assert.assertEquals(1, arvcl.getFiles().length); + Assert.assertTrue((acl.getParent() instanceof VFSClassLoader)); + VFSClassLoader arvcl = (VFSClassLoader) acl.getParent(); + Assert.assertEquals(1, arvcl.getFileObjects().length); // We can't be sure what the authority/host will be due to FQDN mappings, so just check the path - Assert.assertTrue(arvcl.getFiles()[0].getURL().toString().endsWith("/accumulo/classpath/HelloWorld.jar")); + Assert.assertTrue(arvcl.getFileObjects()[0].getURL().toString().contains("/accumulo/classpath/HelloWorld.jar")); Class clazz1 = arvcl.loadClass("test.HelloWorld"); Object o1 = clazz1.newInstance(); Assert.assertEquals("Hello World!", o1.toString()); @@ -110,41 +102,5 @@ public class AccumuloVFSClassLoaderTest hdfs.delete(DEFAULT, true); } - - - @Test - public void testLoadClass() throws Exception { - // Create default and application1 context directory - FileSystem hdfs = cluster.getFileSystem(); - Path DEFAULT = new Path("/accumulo/classpath"); - hdfs.mkdirs(DEFAULT); - Path APPLICATION = new Path("/application1/classpath"); - hdfs.mkdirs(APPLICATION); - - // Copy jar file to DEFAULT and APPLICATION directories - URL jarPath = this.getClass().getResource("/HelloWorld.jar"); - Path src = new Path(jarPath.toURI().toString()); - Path dst = new Path(DEFAULT, src.getName()); - hdfs.copyFromLocalFile(src, dst); - dst = new Path(APPLICATION, src.getName()); - hdfs.copyFromLocalFile(src, dst); - - URL defaultDir = this.getClass().getResource("/application1"); - Configuration conf = new Configuration(hdfs.getConf()); - conf.addResource(new File(defaultDir.getPath() + "/conf/accumulo-site.xml").toURI().toURL()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "ACC_CONF", conf); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "lock", new Object()); - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "log", Logger.getLogger(AccumuloVFSClassLoader.class)); - - Class clazz1 = AccumuloVFSClassLoader.loadClass("test.HelloWorld"); - Object o1 = clazz1.newInstance(); - Assert.assertEquals("Hello World!", o1.toString()); - - Whitebox.setInternalState(AccumuloVFSClassLoader.class, "loader", (AccumuloReloadingVFSClassLoader) null); - - hdfs.delete(DEFAULT, true); - hdfs.delete(APPLICATION, true); - - } } Modified: accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java (original) +++ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java Thu Jan 3 19:18:05 2013 @@ -17,9 +17,7 @@ package org.apache.accumulo.start.classloader.vfs; import java.net.URL; -import java.util.Arrays; import java.util.HashSet; -import java.util.Set; import org.apache.accumulo.start.classloader.vfs.ContextManager.ContextConfig; import org.apache.accumulo.test.AccumuloDFSBase; @@ -61,11 +59,11 @@ public class ContextManagerTest extends ContextManager cm = new ContextManager(vfs, ClassLoader.getSystemClassLoader()); cm.setContextConfig(new ContextConfig() { @Override - public Set getContextURIs(String context) { + public String getContextURIs(String context) { if (context.equals("CX1")) { - return new HashSet(Arrays.asList(new Path(TEST_DIR, "HelloWorld.jar").toUri().toString())); + return new Path(TEST_DIR, "HelloWorld.jar").toUri().toString(); } else if (context.equals("CX2")) { - return new HashSet(Arrays.asList(new Path(TEST_DIR2, "HelloWorld.jar").toUri().toString())); + return new Path(TEST_DIR2, "HelloWorld.jar").toUri().toString(); } return null; } Modified: accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java?rev=1428556&r1=1428462&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java (original) +++ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java Thu Jan 3 19:18:05 2013 @@ -14,11 +14,13 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import java.io.IOException; import java.util.Map; +import org.apache.accumulo.start.classloader.vfs.providers.HdfsFileObject; +import org.apache.accumulo.start.classloader.vfs.providers.ReadOnlyHdfsFileProvider; import org.apache.accumulo.test.AccumuloDFSBase; import org.apache.commons.vfs2.FileObject; import org.apache.commons.vfs2.FileSystemException; Modified: accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java?rev=1428556&r1=1428462&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java (original) +++ accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java Thu Jan 3 19:18:05 2013 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.commons.vfs2.provider; +package org.apache.accumulo.start.classloader.vfs.providers; import java.net.URL; Modified: accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java (original) +++ accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java Thu Jan 3 19:18:05 2013 @@ -21,6 +21,7 @@ import java.io.IOException; import java.io.InputStreamReader; import java.net.URI; +import org.apache.accumulo.start.classloader.vfs.providers.ReadOnlyHdfsFileProvider; import org.apache.commons.vfs2.CacheStrategy; import org.apache.commons.vfs2.FileSystemException; import org.apache.commons.vfs2.cache.DefaultFilesCache; @@ -28,7 +29,6 @@ import org.apache.commons.vfs2.cache.Sof import org.apache.commons.vfs2.impl.DefaultFileReplicator; import org.apache.commons.vfs2.impl.DefaultFileSystemManager; import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory; -import org.apache.commons.vfs2.provider.ReadOnlyHdfsFileProvider; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.MiniDFSCluster; Modified: accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml (original) +++ accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml Thu Jan 3 19:18:05 2013 @@ -101,11 +101,7 @@ - classloader.vfs.enabled - true - - - classloader.vfs.context.classpath.system + general.vfs.classpaths hdfs://localhost:8620/accumulo/classpath Modified: accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml URL: http://svn.apache.org/viewvc/accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml?rev=1428556&r1=1428555&r2=1428556&view=diff ============================================================================== --- accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml (original) +++ accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml Thu Jan 3 19:18:05 2013 @@ -98,11 +98,6 @@ values. - - - - classloader.vfs.enabled - true - +