Return-Path: X-Original-To: apmail-incubator-connectors-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-connectors-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 46CFD7FDF for ; Thu, 15 Dec 2011 16:45:01 +0000 (UTC) Received: (qmail 46312 invoked by uid 500); 15 Dec 2011 16:45:01 -0000 Delivered-To: apmail-incubator-connectors-commits-archive@incubator.apache.org Received: (qmail 46268 invoked by uid 500); 15 Dec 2011 16:45:01 -0000 Mailing-List: contact connectors-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: connectors-dev@incubator.apache.org Delivered-To: mailing list connectors-commits@incubator.apache.org Received: (qmail 46259 invoked by uid 99); 15 Dec 2011 16:45:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Dec 2011 16:45:01 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Dec 2011 16:44:50 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 90E1223889DA; Thu, 15 Dec 2011 16:44:27 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1214850 [1/2] - in /incubator/lcf/trunk: ./ framework/ framework/example-common/ framework/example-multiprocess/ framework/example-singleprocess/ framework/jetty-example/ framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunn... Date: Thu, 15 Dec 2011 16:44:27 -0000 To: connectors-commits@incubator.apache.org From: kwright@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111215164427.90E1223889DA@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: kwright Date: Thu Dec 15 16:44:26 2011 New Revision: 1214850 URL: http://svn.apache.org/viewvc?rev=1214850&view=rev Log: Fix for CONNECTORS-313. Added: incubator/lcf/trunk/framework/example-common/ - copied from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-common/ incubator/lcf/trunk/framework/example-common/connectors.xml - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-common/connectors.xml incubator/lcf/trunk/framework/example-multiprocess/ - copied from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/ incubator/lcf/trunk/framework/example-multiprocess/initialize.bat - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/initialize.bat incubator/lcf/trunk/framework/example-multiprocess/initialize.sh - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/initialize.sh incubator/lcf/trunk/framework/example-multiprocess/lock-clean.bat - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/lock-clean.bat incubator/lcf/trunk/framework/example-multiprocess/lock-clean.sh - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/lock-clean.sh incubator/lcf/trunk/framework/example-multiprocess/logging.ini - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/logging.ini incubator/lcf/trunk/framework/example-multiprocess/properties.xml - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/properties.xml incubator/lcf/trunk/framework/example-multiprocess/start-agents.bat - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/start-agents.bat incubator/lcf/trunk/framework/example-multiprocess/start-agents.sh - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/start-agents.sh incubator/lcf/trunk/framework/example-multiprocess/start-database.bat - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/start-database.bat incubator/lcf/trunk/framework/example-multiprocess/start-database.sh - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/start-database.sh incubator/lcf/trunk/framework/example-multiprocess/stop-agents.bat - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/stop-agents.bat incubator/lcf/trunk/framework/example-multiprocess/stop-agents.sh - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-multiprocess/stop-agents.sh incubator/lcf/trunk/framework/example-singleprocess/ - copied from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-singleprocess/ incubator/lcf/trunk/framework/example-singleprocess/logging.ini - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-singleprocess/logging.ini incubator/lcf/trunk/framework/example-singleprocess/properties.xml - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/example-singleprocess/properties.xml incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/InitializeAndRegister.java - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/InitializeAndRegister.java incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/interfaces/Connectors.java - copied unchanged from r1214848, incubator/lcf/branches/CONNECTORS-313/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/interfaces/Connectors.java Removed: incubator/lcf/trunk/framework/jetty-example/ incubator/lcf/trunk/framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunner/Connectors.java Modified: incubator/lcf/trunk/ (props changed) incubator/lcf/trunk/CHANGES.txt incubator/lcf/trunk/build.xml incubator/lcf/trunk/framework/build.xml incubator/lcf/trunk/framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunner/ManifoldCFJettyRunner.java incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/ManifoldCF.java incubator/lcf/trunk/site/src/documentation/content/xdocs/how-to-build-and-deploy.xml Propchange: incubator/lcf/trunk/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Dec 15 16:44:26 2011 @@ -12,4 +12,5 @@ /incubator/lcf/branches/CONNECTORS-284:1189305-1190398 /incubator/lcf/branches/CONNECTORS-287:1200390-1210394 /incubator/lcf/branches/CONNECTORS-290:1204836-1205502 +/incubator/lcf/branches/CONNECTORS-313:1213173-1214848 /incubator/lcf/branches/CONNECTORS-32:1092556-1094216 Modified: incubator/lcf/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/incubator/lcf/trunk/CHANGES.txt?rev=1214850&r1=1214849&r2=1214850&view=diff ============================================================================== --- incubator/lcf/trunk/CHANGES.txt (original) +++ incubator/lcf/trunk/CHANGES.txt Thu Dec 15 16:44:26 2011 @@ -3,6 +3,10 @@ $Id$ ======================= 0.4-dev ===================== +CONNECTORS-313: Reorganize the dist directory to include a multiprocess +example as well as a single process example. +(Shinichiro Abe, Karl Wright) + CONNECTORS-320: Fix issues with remote HSQLDB driver having to do with schema discovery. (Karl Wright) Modified: incubator/lcf/trunk/build.xml URL: http://svn.apache.org/viewvc/incubator/lcf/trunk/build.xml?rev=1214850&r1=1214849&r2=1214850&view=diff ============================================================================== --- incubator/lcf/trunk/build.xml (original) +++ incubator/lcf/trunk/build.xml Thu Dec 15 16:44:26 2011 @@ -1178,19 +1178,43 @@ - - - - + + + + + + + + + + + + + + + + + - - - + + + + + + + + + + + + + @@ -1215,10 +1239,11 @@ - - + + + @@ -1253,10 +1278,11 @@ - - + + + @@ -1281,10 +1307,12 @@ - - + + + + @@ -1309,8 +1337,8 @@ - - + + @@ -1321,6 +1349,8 @@ + + @@ -1345,8 +1375,8 @@ - - + + @@ -1357,6 +1387,7 @@ + @@ -1381,10 +1412,11 @@ - - + + + @@ -1409,10 +1441,11 @@ - - + + + @@ -1437,10 +1470,11 @@ - - + + + @@ -1465,10 +1499,11 @@ - - + + + @@ -1493,10 +1528,12 @@ - - + + + + @@ -1521,14 +1558,15 @@ - - + + + @@ -1553,10 +1591,12 @@ - - + + + + @@ -1581,8 +1621,8 @@ - - + + @@ -1593,6 +1633,10 @@ + + + + @@ -1617,10 +1661,11 @@ - - + + + @@ -1645,10 +1690,11 @@ - - + + + @@ -1673,10 +1719,11 @@ - - + + + @@ -1701,10 +1748,11 @@ - - + + + @@ -1729,8 +1777,8 @@ - - + + @@ -1741,6 +1789,9 @@ + + + @@ -1765,10 +1816,11 @@ - - + + + @@ -1793,10 +1845,11 @@ - - + + + @@ -1806,221 +1859,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -3651,7 +3489,7 @@ - + Modified: incubator/lcf/trunk/framework/build.xml URL: http://svn.apache.org/viewvc/incubator/lcf/trunk/framework/build.xml?rev=1214850&r1=1214849&r2=1214850&view=diff ============================================================================== --- incubator/lcf/trunk/framework/build.xml (original) +++ incubator/lcf/trunk/framework/build.xml Thu Dec 15 16:44:26 2011 @@ -333,24 +333,39 @@ + + + + + + + + - - + + - - + + - - + + - - - + + + + + + + + + + @@ -373,20 +388,30 @@ - - - - + + + + + + + + + + + + + + @@ -398,8 +423,6 @@ - - @@ -413,7 +436,7 @@ - + @@ -444,8 +467,6 @@ - - @@ -456,15 +477,15 @@ - + - - + - + + @@ -681,7 +702,7 @@ - + Modified: incubator/lcf/trunk/framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunner/ManifoldCFJettyRunner.java URL: http://svn.apache.org/viewvc/incubator/lcf/trunk/framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunner/ManifoldCFJettyRunner.java?rev=1214850&r1=1214849&r2=1214850&view=diff ============================================================================== --- incubator/lcf/trunk/framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunner/ManifoldCFJettyRunner.java (original) +++ incubator/lcf/trunk/framework/jetty-runner/src/main/java/org/apache/manifoldcf/jettyrunner/ManifoldCFJettyRunner.java Thu Dec 15 16:44:26 2011 @@ -19,10 +19,8 @@ package org.apache.manifoldcf.jettyrunne import java.io.*; import org.apache.manifoldcf.core.interfaces.*; -import org.apache.manifoldcf.agents.interfaces.*; -import org.apache.manifoldcf.crawler.interfaces.*; -import org.apache.manifoldcf.authorities.interfaces.*; -import org.apache.manifoldcf.agents.system.*; +import org.apache.manifoldcf.crawler.system.*; +import org.apache.manifoldcf.crawler.*; import java.io.IOException; import java.io.InputStream; @@ -50,18 +48,6 @@ public class ManifoldCFJettyRunner public static final String _rcsid = "@(#)$Id: ManifoldCFJettyRunner.java 989983 2010-08-27 00:10:12Z kwright $"; public static final String agentShutdownSignal = org.apache.manifoldcf.agents.AgentRun.agentShutdownSignal; - - // Configuration parameters - public static final String connectorsConfigurationFile = "org.apache.manifoldcf.connectorsconfigurationfile"; - public static final String databaseSuperuserName = "org.apache.manifoldcf.dbsuperusername"; - public static final String databaseSuperuserPassword = "org.apache.manifoldcf.dbsuperuserpassword"; - - // Connectors configuration file - public static final String NODE_OUTPUTCONNECTOR = "outputconnector"; - public static final String NODE_AUTHORITYCONNECTOR = "authorityconnector"; - public static final String NODE_REPOSITORYCONNECTOR = "repositoryconnector"; - public static final String ATTRIBUTE_NAME = "name"; - public static final String ATTRIBUTE_CLASS = "class"; protected Server server; @@ -138,270 +124,6 @@ public class ManifoldCFJettyRunner return conns[0].getLocalPort(); } - /** Clear the agents shutdown signal */ - public static void clearShutdownSignal(IThreadContext tc) - throws ManifoldCFException - { - // Clear the agents shutdown signal. - ILockManager lockManager = LockManagerFactory.make(tc); - lockManager.clearGlobalFlag(agentShutdownSignal); - } - - /** Create the database and the schema */ - public static void createDatabaseAndSchema(IThreadContext tc, String superuserName, String superuserPassword) - throws ManifoldCFException - { - ManifoldCF.createSystemDatabase(tc,superuserName,superuserPassword); - ManifoldCF.installTables(tc); - } - - /** Register the agents */ - public static void registerAgents(IThreadContext tc) - throws ManifoldCFException - { - // Register - IAgentManager agentMgr = AgentManagerFactory.make(tc); - agentMgr.registerAgent("org.apache.manifoldcf.crawler.system.CrawlerAgent"); - } - - /** Read connectors configuration file. - */ - public static Connectors readConnectorDeclarations(File connectorConfigFile) - throws ManifoldCFException - { - Connectors c = null; - if (connectorConfigFile != null) - { - try - { - // Open the file, read it, and attempt to do the connector registrations - InputStream is = new FileInputStream(connectorConfigFile); - try - { - c = new Connectors(is); - } - finally - { - is.close(); - } - } - catch (FileNotFoundException e) - { - throw new ManifoldCFException("Couldn't find connector configuration file: "+e.getMessage(),e); - } - catch (IOException e) - { - throw new ManifoldCFException("Error reading connector configuration file: "+e.getMessage(),e); - } - } - return c; - } - - /** Unregister all connectors. - */ - public static void unregisterAllConnectors(IThreadContext tc) - throws ManifoldCFException - { - // Grab a database handle, so we can use transactions later. - IDBInterface database = DBInterfaceFactory.make(tc, - ManifoldCF.getMasterDatabaseName(), - ManifoldCF.getMasterDatabaseUsername(), - ManifoldCF.getMasterDatabasePassword()); - - // Output connectors... - { - IOutputConnectorManager mgr = OutputConnectorManagerFactory.make(tc); - IOutputConnectionManager connManager = OutputConnectionManagerFactory.make(tc); - IResultSet classNames = mgr.getConnectors(); - int i = 0; - while (i < classNames.getRowCount()) - { - IResultRow row = classNames.getRow(i++); - String className = (String)row.getValue("classname"); - // Deregistration should be done in a transaction - database.beginTransaction(); - try - { - // Find the connection names that come with this class - String[] connectionNames = connManager.findConnectionsForConnector(className); - // For all connection names, notify all agents of the deregistration - AgentManagerFactory.noteOutputConnectorDeregistration(tc,connectionNames); - // Now that all jobs have been placed into an appropriate state, actually do the deregistration itself. - mgr.unregisterConnector(className); - } - catch (ManifoldCFException e) - { - database.signalRollback(); - throw e; - } - catch (Error e) - { - database.signalRollback(); - throw e; - } - finally - { - database.endTransaction(); - } - } - System.err.println("Successfully unregistered all output connectors"); - } - - // Authority connectors... - { - IAuthorityConnectorManager mgr = AuthorityConnectorManagerFactory.make(tc); - IResultSet classNames = mgr.getConnectors(); - int i = 0; - while (i < classNames.getRowCount()) - { - IResultRow row = classNames.getRow(i++); - mgr.unregisterConnector((String)row.getValue("classname")); - } - System.err.println("Successfully unregistered all authority connectors"); - } - - // Repository connectors... - { - IConnectorManager mgr = ConnectorManagerFactory.make(tc); - IJobManager jobManager = JobManagerFactory.make(tc); - IRepositoryConnectionManager connManager = RepositoryConnectionManagerFactory.make(tc); - IResultSet classNames = mgr.getConnectors(); - int i = 0; - while (i < classNames.getRowCount()) - { - IResultRow row = classNames.getRow(i++); - String className = (String)row.getValue("classname"); - // Deregistration should be done in a transaction - database.beginTransaction(); - try - { - // Find the connection names that come with this class - String[] connectionNames = connManager.findConnectionsForConnector(className); - // For each connection name, modify the jobs to note that the connector is no longer installed - jobManager.noteConnectorDeregistration(connectionNames); - // Now that all jobs have been placed into an appropriate state, actually do the deregistration itself. - mgr.unregisterConnector(className); - } - catch (ManifoldCFException e) - { - database.signalRollback(); - throw e; - } - catch (Error e) - { - database.signalRollback(); - throw e; - } - finally - { - database.endTransaction(); - } - } - System.err.println("Successfully unregistered all repository connectors"); - } - } - - /** Register all connectors as specified by a Connectors structure, usually read from the connectors.xml file. - */ - public static void registerConnectors(IThreadContext tc, Connectors c) - throws ManifoldCFException - { - if (c != null) - { - // Grab a database handle, so we can use transactions later. - IDBInterface database = DBInterfaceFactory.make(tc, - ManifoldCF.getMasterDatabaseName(), - ManifoldCF.getMasterDatabaseUsername(), - ManifoldCF.getMasterDatabasePassword()); - - // Other code will go here to discover and register various connectors that exist in the classpath - int i = 0; - while (i < c.getChildCount()) - { - ConfigurationNode cn = c.findChild(i++); - if (cn.getType().equals(NODE_OUTPUTCONNECTOR)) - { - String name = cn.getAttributeValue(ATTRIBUTE_NAME); - String className = cn.getAttributeValue(ATTRIBUTE_CLASS); - IOutputConnectorManager mgr = OutputConnectorManagerFactory.make(tc); - IOutputConnectionManager connManager = OutputConnectionManagerFactory.make(tc); - // Registration should be done in a transaction - database.beginTransaction(); - try - { - // First, register connector - mgr.registerConnector(name,className); - // Then, signal to all jobs that might depend on this connector that they can switch state - // Find the connection names that come with this class - String[] connectionNames = connManager.findConnectionsForConnector(className); - // For all connection names, notify all agents of the registration - AgentManagerFactory.noteOutputConnectorRegistration(tc,connectionNames); - } - catch (ManifoldCFException e) - { - database.signalRollback(); - throw e; - } - catch (Error e) - { - database.signalRollback(); - throw e; - } - finally - { - database.endTransaction(); - } - System.err.println("Successfully registered output connector '"+className+"'"); - } - else if (cn.getType().equals(NODE_AUTHORITYCONNECTOR)) - { - String name = cn.getAttributeValue(ATTRIBUTE_NAME); - String className = cn.getAttributeValue(ATTRIBUTE_CLASS); - IAuthorityConnectorManager mgr = AuthorityConnectorManagerFactory.make(tc); - mgr.registerConnector(name,className); - System.err.println("Successfully registered authority connector '"+className+"'"); - } - else if (cn.getType().equals(NODE_REPOSITORYCONNECTOR)) - { - String name = cn.getAttributeValue(ATTRIBUTE_NAME); - String className = cn.getAttributeValue(ATTRIBUTE_CLASS); - IConnectorManager mgr = ConnectorManagerFactory.make(tc); - IJobManager jobManager = JobManagerFactory.make(tc); - IRepositoryConnectionManager connManager = RepositoryConnectionManagerFactory.make(tc); - // Deregistration should be done in a transaction - database.beginTransaction(); - try - { - // First, register connector - mgr.registerConnector(name,className); - // Then, signal to all jobs that might depend on this connector that they can switch state - // Find the connection names that come with this class - String[] connectionNames = connManager.findConnectionsForConnector(className); - // For each connection name, modify the jobs to note that the connector is now installed - jobManager.noteConnectorRegistration(connectionNames); - } - catch (ManifoldCFException e) - { - database.signalRollback(); - throw e; - } - catch (Error e) - { - database.signalRollback(); - throw e; - } - finally - { - database.endTransaction(); - } - System.err.println("Successfully registered repository connector '"+className+"'"); - } - else - throw new ManifoldCFException("Unrecognized connectors node type '"+cn.getType()+"'"); - } - } - } - /** Run the agents process. This method will not return unless the agents process is shut down. */ public static void runAgents(IThreadContext tc) @@ -472,30 +194,18 @@ public class ManifoldCFJettyRunner ManifoldCF.initializeEnvironment(); IThreadContext tc = ThreadContextFactory.make(); - // Clear the shutdown signal - clearShutdownSignal(tc); - - // Get the specified superuser name and password, in case this isn't Derby we're using - String superuserName = ManifoldCF.getProperty(databaseSuperuserName); - if (superuserName == null) - superuserName = ""; - String superuserPassword = ManifoldCF.getProperty(databaseSuperuserPassword); - if (superuserPassword == null) - superuserPassword = ""; + // Clear the agents shutdown signal. + ILockManager lockManager = LockManagerFactory.make(tc); + lockManager.clearGlobalFlag(agentShutdownSignal); // Do the basic initialization of the database and its schema - createDatabaseAndSchema(tc,superuserName,superuserPassword); - registerAgents(tc); + ManifoldCF.createSystemDatabase(tc); - // Read connectors configuration file (to figure out what we need to register) - File connectorConfigFile = ManifoldCF.getFileProperty(connectorsConfigurationFile); - Connectors c = readConnectorDeclarations(connectorConfigFile); + ManifoldCF.installTables(tc); - // Unregister all connectors. - unregisterAllConnectors(tc); - - // Register connections specified by connectors.xml - registerConnectors(tc,c); + org.apache.manifoldcf.crawler.system.ManifoldCF.registerThisAgent(tc); + + ManifoldCF.reregisterAllConnectors(tc); System.err.println("Starting jetty..."); Modified: incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/ManifoldCF.java URL: http://svn.apache.org/viewvc/incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/ManifoldCF.java?rev=1214850&r1=1214849&r2=1214850&view=diff ============================================================================== --- incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/ManifoldCF.java (original) +++ incubator/lcf/trunk/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/ManifoldCF.java Thu Dec 15 16:44:26 2011 @@ -75,12 +75,16 @@ public class ManifoldCF extends org.apac // Factor in amount to stuff protected static float stuffAmtFactor = 0.5f; + // Properties protected static final String workerThreadCountProperty = "org.apache.manifoldcf.crawler.threads"; protected static final String deleteThreadCountProperty = "org.apache.manifoldcf.crawler.deletethreads"; protected static final String cleanupThreadCountProperty = "org.apache.manifoldcf.crawler.cleanupthreads"; protected static final String expireThreadCountProperty = "org.apache.manifoldcf.crawler.expirethreads"; protected static final String lowWaterFactorProperty = "org.apache.manifoldcf.crawler.lowwaterfactor"; protected static final String stuffAmtFactorProperty = "org.apache.manifoldcf.crawler.stuffamountfactor"; + protected static final String connectorsConfigurationFileProperty = "org.apache.manifoldcf.connectorsconfigurationfile"; + protected static final String databaseSuperuserNameProperty = "org.apache.manifoldcf.dbsuperusername"; + protected static final String databaseSuperuserPasswordProperty = "org.apache.manifoldcf.dbsuperuserpassword"; /** This object is used to make sure the initialization sequence is atomic. Shutdown cannot occur until the system is in a known state. */ protected static Integer startupLock = new Integer(0); @@ -104,6 +108,290 @@ public class ManifoldCF extends org.apac } } + /** Create system database using superuser properties from properties.xml. + */ + public static void createSystemDatabase(IThreadContext threadContext) + throws ManifoldCFException + { + // Get the specified superuser name and password, in case this isn't Derby we're using + String superuserName = getProperty(databaseSuperuserNameProperty); + if (superuserName == null) + superuserName = ""; + String superuserPassword = getProperty(databaseSuperuserPasswordProperty); + if (superuserPassword == null) + superuserPassword = ""; + createSystemDatabase(threadContext,superuserName,superuserPassword); + } + + /** Register this agent */ + public static void registerThisAgent(IThreadContext tc) + throws ManifoldCFException + { + // Register + IAgentManager agentMgr = AgentManagerFactory.make(tc); + agentMgr.registerAgent("org.apache.manifoldcf.crawler.system.CrawlerAgent"); + } + + /** Register or re-register all connectors, based on a connectors.xml file. + */ + public static void reregisterAllConnectors(IThreadContext tc) + throws ManifoldCFException + { + // Read connectors configuration file (to figure out what we need to register) + File connectorConfigFile = getFileProperty(connectorsConfigurationFileProperty); + Connectors c = readConnectorDeclarations(connectorConfigFile); + + // Unregister all connectors. + unregisterAllConnectors(tc); + + // Register connections specified by connectors.xml + registerConnectors(tc,c); + } + + /** Read connectors configuration file. + */ + public static Connectors readConnectorDeclarations(File connectorConfigFile) + throws ManifoldCFException + { + Connectors c = null; + if (connectorConfigFile != null) + { + try + { + // Open the file, read it, and attempt to do the connector registrations + InputStream is = new FileInputStream(connectorConfigFile); + try + { + c = new Connectors(is); + } + finally + { + is.close(); + } + } + catch (FileNotFoundException e) + { + throw new ManifoldCFException("Couldn't find connector configuration file: "+e.getMessage(),e); + } + catch (IOException e) + { + throw new ManifoldCFException("Error reading connector configuration file: "+e.getMessage(),e); + } + } + return c; + } + + /** Unregister all connectors. + */ + public static void unregisterAllConnectors(IThreadContext tc) + throws ManifoldCFException + { + // Grab a database handle, so we can use transactions later. + IDBInterface database = DBInterfaceFactory.make(tc, + ManifoldCF.getMasterDatabaseName(), + ManifoldCF.getMasterDatabaseUsername(), + ManifoldCF.getMasterDatabasePassword()); + + // Output connectors... + { + IOutputConnectorManager mgr = OutputConnectorManagerFactory.make(tc); + IOutputConnectionManager connManager = OutputConnectionManagerFactory.make(tc); + IResultSet classNames = mgr.getConnectors(); + int i = 0; + while (i < classNames.getRowCount()) + { + IResultRow row = classNames.getRow(i++); + String className = (String)row.getValue("classname"); + // Deregistration should be done in a transaction + database.beginTransaction(); + try + { + // Find the connection names that come with this class + String[] connectionNames = connManager.findConnectionsForConnector(className); + // For all connection names, notify all agents of the deregistration + AgentManagerFactory.noteOutputConnectorDeregistration(tc,connectionNames); + // Now that all jobs have been placed into an appropriate state, actually do the deregistration itself. + mgr.unregisterConnector(className); + } + catch (ManifoldCFException e) + { + database.signalRollback(); + throw e; + } + catch (Error e) + { + database.signalRollback(); + throw e; + } + finally + { + database.endTransaction(); + } + } + System.err.println("Successfully unregistered all output connectors"); + } + + // Authority connectors... + { + IAuthorityConnectorManager mgr = AuthorityConnectorManagerFactory.make(tc); + IResultSet classNames = mgr.getConnectors(); + int i = 0; + while (i < classNames.getRowCount()) + { + IResultRow row = classNames.getRow(i++); + mgr.unregisterConnector((String)row.getValue("classname")); + } + System.err.println("Successfully unregistered all authority connectors"); + } + + // Repository connectors... + { + IConnectorManager mgr = ConnectorManagerFactory.make(tc); + IJobManager jobManager = JobManagerFactory.make(tc); + IRepositoryConnectionManager connManager = RepositoryConnectionManagerFactory.make(tc); + IResultSet classNames = mgr.getConnectors(); + int i = 0; + while (i < classNames.getRowCount()) + { + IResultRow row = classNames.getRow(i++); + String className = (String)row.getValue("classname"); + // Deregistration should be done in a transaction + database.beginTransaction(); + try + { + // Find the connection names that come with this class + String[] connectionNames = connManager.findConnectionsForConnector(className); + // For each connection name, modify the jobs to note that the connector is no longer installed + jobManager.noteConnectorDeregistration(connectionNames); + // Now that all jobs have been placed into an appropriate state, actually do the deregistration itself. + mgr.unregisterConnector(className); + } + catch (ManifoldCFException e) + { + database.signalRollback(); + throw e; + } + catch (Error e) + { + database.signalRollback(); + throw e; + } + finally + { + database.endTransaction(); + } + } + System.err.println("Successfully unregistered all repository connectors"); + } + } + + // Connectors configuration file + protected static final String NODE_OUTPUTCONNECTOR = "outputconnector"; + protected static final String NODE_AUTHORITYCONNECTOR = "authorityconnector"; + protected static final String NODE_REPOSITORYCONNECTOR = "repositoryconnector"; + protected static final String ATTRIBUTE_NAME = "name"; + protected static final String ATTRIBUTE_CLASS = "class"; + + /** Register all connectors as specified by a Connectors structure, usually read from the connectors.xml file. + */ + public static void registerConnectors(IThreadContext tc, Connectors c) + throws ManifoldCFException + { + if (c != null) + { + // Grab a database handle, so we can use transactions later. + IDBInterface database = DBInterfaceFactory.make(tc, + ManifoldCF.getMasterDatabaseName(), + ManifoldCF.getMasterDatabaseUsername(), + ManifoldCF.getMasterDatabasePassword()); + + // Other code will go here to discover and register various connectors that exist in the classpath + int i = 0; + while (i < c.getChildCount()) + { + ConfigurationNode cn = c.findChild(i++); + if (cn.getType().equals(NODE_OUTPUTCONNECTOR)) + { + String name = cn.getAttributeValue(ATTRIBUTE_NAME); + String className = cn.getAttributeValue(ATTRIBUTE_CLASS); + IOutputConnectorManager mgr = OutputConnectorManagerFactory.make(tc); + IOutputConnectionManager connManager = OutputConnectionManagerFactory.make(tc); + // Registration should be done in a transaction + database.beginTransaction(); + try + { + // First, register connector + mgr.registerConnector(name,className); + // Then, signal to all jobs that might depend on this connector that they can switch state + // Find the connection names that come with this class + String[] connectionNames = connManager.findConnectionsForConnector(className); + // For all connection names, notify all agents of the registration + AgentManagerFactory.noteOutputConnectorRegistration(tc,connectionNames); + } + catch (ManifoldCFException e) + { + database.signalRollback(); + throw e; + } + catch (Error e) + { + database.signalRollback(); + throw e; + } + finally + { + database.endTransaction(); + } + System.err.println("Successfully registered output connector '"+className+"'"); + } + else if (cn.getType().equals(NODE_AUTHORITYCONNECTOR)) + { + String name = cn.getAttributeValue(ATTRIBUTE_NAME); + String className = cn.getAttributeValue(ATTRIBUTE_CLASS); + IAuthorityConnectorManager mgr = AuthorityConnectorManagerFactory.make(tc); + mgr.registerConnector(name,className); + System.err.println("Successfully registered authority connector '"+className+"'"); + } + else if (cn.getType().equals(NODE_REPOSITORYCONNECTOR)) + { + String name = cn.getAttributeValue(ATTRIBUTE_NAME); + String className = cn.getAttributeValue(ATTRIBUTE_CLASS); + IConnectorManager mgr = ConnectorManagerFactory.make(tc); + IJobManager jobManager = JobManagerFactory.make(tc); + IRepositoryConnectionManager connManager = RepositoryConnectionManagerFactory.make(tc); + // Deregistration should be done in a transaction + database.beginTransaction(); + try + { + // First, register connector + mgr.registerConnector(name,className); + // Then, signal to all jobs that might depend on this connector that they can switch state + // Find the connection names that come with this class + String[] connectionNames = connManager.findConnectionsForConnector(className); + // For each connection name, modify the jobs to note that the connector is now installed + jobManager.noteConnectorRegistration(connectionNames); + } + catch (ManifoldCFException e) + { + database.signalRollback(); + throw e; + } + catch (Error e) + { + database.signalRollback(); + throw e; + } + finally + { + database.endTransaction(); + } + System.err.println("Successfully registered repository connector '"+className+"'"); + } + else + throw new ManifoldCFException("Unrecognized connectors node type '"+cn.getType()+"'"); + } + } + } /** Install all the crawler system tables. *@param threadcontext is the thread context.