From droids-commits-return-484-apmail-incubator-droids-commits-archive=incubator.apache.org@incubator.apache.org Fri Dec 7 17:14:07 2012 Return-Path: X-Original-To: apmail-incubator-droids-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-droids-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5CE45ECD4 for ; Fri, 7 Dec 2012 17:14:07 +0000 (UTC) Received: (qmail 89153 invoked by uid 500); 7 Dec 2012 17:14:07 -0000 Delivered-To: apmail-incubator-droids-commits-archive@incubator.apache.org Received: (qmail 88906 invoked by uid 500); 7 Dec 2012 17:14:06 -0000 Mailing-List: contact droids-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: droids-dev@incubator.apache.org Delivered-To: mailing list droids-commits@incubator.apache.org Received: (qmail 88898 invoked by uid 99); 7 Dec 2012 17:14:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Dec 2012 17:14:06 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Dec 2012 17:13:55 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 4DD97238897F; Fri, 7 Dec 2012 17:13:32 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1418399 [1/2] - in /incubator/droids/branches/0.2.x-cleanup: ./ droids-core/src/main/java/org/apache/droids/core/ droids-core/src/main/java/org/apache/droids/filter/ droids-core/src/main/java/org/apache/droids/handle/ droids-core/src/main/... Date: Fri, 07 Dec 2012 17:13:15 -0000 To: droids-commits@incubator.apache.org From: tobr@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20121207171332.4DD97238897F@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: tobr Date: Fri Dec 7 17:12:35 2012 New Revision: 1418399 URL: http://svn.apache.org/viewvc?rev=1418399&view=rev Log: hide factory calls from API simplified API Added: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java (with props) incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Parser.java - copied, changed from r1417006, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/parse/Parser.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/FilterFactory.java - copied, changed from r1417023, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/URLFiltersFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/util/ - copied from r1417006, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/util/ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java (contents, props changed) - copied, changed from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingDroid.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingWorker.java (contents, props changed) - copied, changed from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingWorker.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/Link.java (contents, props changed) - copied, changed from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/api/Link.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/LinkTask.java (contents, props changed) - copied, changed from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/LinkTask.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java (contents, props changed) - copied, changed from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/ReportCrawlingDroid.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/ - copied from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/protocol/http/ incubator/droids/branches/0.2.x-cleanup/droids-examples/src/main/java/org/apache/droids/localserver/ - copied from r1417006, incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/localserver/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/protocol/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/protocol/file/ - copied from r1417006, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/protocol/file/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/walker/ - copied from r1417006, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/walker/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/java/org/apache/droids/walker/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/java/org/apache/droids/walker/WalkingDroidTest.java (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-apache-cloudStack-scalability.odp (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-building-cross-platform-hybrid-applications-using-AMQP-1_0-with-apache-qpid.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-building-cross-platform-hybrid-applications-using-AMQP-1_0-with-apache-qpid.pptx (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-getting-started-with-AMQP-1_0-using-apache-qpid.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-integration-in-the-cloud-IPaaS-with-fuse-technology.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-introduction-to-apache-cloudstack.odp (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/L2R_Cloud/aceu-2012-what-is-new-in-cloudstack-4.0.odp (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/ incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-elastic-search-in-production_lessons-learned.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-fundamentals-of-information-retrieval-illustration-with-apache-lucene.pptx (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-lucene-4-performance-tuning.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-query-parsing-tips-and-tricks.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-searching-relational-like-data-with-lucene.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-solr-4-the-NoSQL-database.pdf (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-solr-4-the-NoSQL-database.pptx (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/PR-Lucene/aceu-2012-solrcloud-round-table.pptx (with props) incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/resources/docs/aceu-2012-apachecon-europe-keynote.pdf (with props) Removed: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/DroidFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/GenericFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/URLFiltersFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/parse/ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/protocol/ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/App.java incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/App.java incubator/droids/branches/0.2.x-cleanup/droids-walker/src/test/java/org/apache/droids/AppTest.java Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/AbstractDroid.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Droid.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Handler.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/MultiThreadedTaskMaster.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Protocol.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Task.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Worker.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/AlreadyVisitedFilter.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/ChainTaskFilter.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/Filter.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/HostFilter.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/MaxDepthTaskFilter.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/RegexURLFilter.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/handle/SaveHandler.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/handle/SysoutHandler.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/HandlerFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ParserFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ProtocolFactory.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/SimpleTask.java incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/TestSimpleQueue.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/pom.xml incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpContentEntity.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpProtocol.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpContentEntity.java incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpProtocol.java incubator/droids/branches/0.2.x-cleanup/droids-spring/src/main/resources/org/apache/droids/dynamic/droids-core-context.xml incubator/droids/branches/0.2.x-cleanup/droids-spring/src/main/resources/org/apache/droids/dynamic/droids-core-factories-context.xml incubator/droids/branches/0.2.x-cleanup/droids-spring/src/test/resources/droids-core-test-context.xml incubator/droids/branches/0.2.x-cleanup/droids-walker/pom.xml incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/protocol/file/FileProtocol.java incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/walker/FileTask.java incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/walker/FileWorker.java incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/walker/SimpleWalkingDroid.java incubator/droids/branches/0.2.x-cleanup/droids-walker/src/main/java/org/apache/droids/walker/WalkingDroid.java incubator/droids/branches/0.2.x-cleanup/pom.xml Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/AbstractDroid.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/AbstractDroid.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/AbstractDroid.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/AbstractDroid.java Fri Dec 7 17:12:35 2012 @@ -16,42 +16,94 @@ */ package org.apache.droids.core; +import org.apache.droids.helper.factories.FilterFactory; +import org.apache.droids.helper.factories.HandlerFactory; +import org.apache.droids.helper.factories.ParserFactory; +import org.apache.droids.helper.factories.ProtocolFactory; +import org.apache.droids.filter.Filter; + +import java.io.IOException; import java.util.Queue; import java.util.concurrent.TimeUnit; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - /** * Manage common tasks in standard Droids */ public abstract class AbstractDroid implements Droid { + protected final Queue queue; + protected final TaskMaster taskMaster; + protected ProtocolFactory protocolFactory; + protected ParserFactory parserFactory; + protected FilterFactory filterFactory; + protected HandlerFactory handlerFactory; + + public AbstractDroid() { + this.queue = new SimpleTaskQueueWithHistory(); + this.taskMaster = new MultiThreadedTaskMaster(); + this.protocolFactory = new ProtocolFactory(); + this.parserFactory = new ParserFactory(); + this.filterFactory = new FilterFactory(); + this.handlerFactory = new HandlerFactory(); + } + + public AbstractDroid(Queue queue, TaskMaster taskMaster) { + this.queue = queue; + this.taskMaster = taskMaster; + } + + @Override + public void start() { + taskMaster.start(queue, this); + try { + taskMaster.awaitTermination(1000, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + + @Override + public void add(T task) { + queue.add(task); + } + + @Override + public void parse(T task) throws DroidsException, IOException { + this.parserFactory.parse(task); + } + + @Override + public void handle(T task) throws DroidsException, IOException { + this.handlerFactory.handle(task); + } + + @Override + public T filter(T task) { + return this.filterFactory.filter(task); + } + + public void addParsers(Parser... parsers) { + for (Parser parser : parsers) { + this.parserFactory.addParser(parser); + } + } + + public void addProtocols(Protocol... protocols) { + for (Protocol protocol : protocols) { + this.protocolFactory.addProtocol(protocol); + } + } + + public void addHandlers(Handler... handlers) { + for (Handler handler : handlers) { + this.handlerFactory.addHandler(handler); + } + } + + public void addFilters(Filter... filters) { + for (Filter filter : filters) { + this.filterFactory.addFilter(filter); + } + } + - protected final Logger log = LoggerFactory.getLogger(AbstractDroid.class); - protected final Queue queue; - protected final TaskMaster taskMaster; - - public AbstractDroid(Queue queue, TaskMaster taskMaster) { - this.queue = queue; - this.taskMaster = taskMaster; - } - - @Override - public void start() { - taskMaster.start(queue, this); - try { - taskMaster.awaitTermination(1000, TimeUnit.MILLISECONDS); - } catch (InterruptedException e) { - e.printStackTrace(); - } - } - - public Queue getQueue() { - return queue; - } - - @Override - public TaskMaster getTaskMaster() { - return taskMaster; - } } Added: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java?rev=1418399&view=auto ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java (added) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java Fri Dec 7 17:12:35 2012 @@ -0,0 +1,31 @@ +package org.apache.droids.core; + +import java.util.HashMap; +import java.util.Map; + +/** + * + * + * + */ +public class ContentEntity { + private Map data; + + public final static String CONTENT = "content"; + public final static String MIME_TYPE = "mime"; + public final static String CONTENT_LENGTH = "content-length"; + + public ContentEntity() { + this.data = new HashMap(); + } + + public Object getValue(String key) { + return data.get(key); + } + + public void put(String key, Object value) { + this.data.put(key, value); + } + + +} Propchange: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java ------------------------------------------------------------------------------ svn:keywords = Author Date Id Revision Propchange: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/ContentEntity.java ------------------------------------------------------------------------------ svn:mime-type = text/plain Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Droid.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Droid.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Droid.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Droid.java Fri Dec 7 17:12:35 2012 @@ -16,57 +16,61 @@ */ package org.apache.droids.core; -import java.util.Queue; +import java.io.IOException; /** * Interface for a droid. Droid can be seen as a "project manger" that delegates * the work to {@link Worker} units. - *

+ *

* Droids aims to be an intelligent standalone robot framework that allows to * create and extend existing droids (robots). In the future it will offer an - * administration application to manage and controll the different droids. - * + * administration application to manage and control the different droids. + * * @version 1.0 */ public interface Droid { - - /** - * Initialize the queue. Can have different implementation but the main - * groups normally are - *

    - *
  1. add only one url, from which we then start crawling
    2. - *
  2. add an array of start urls and then crawl them
    3. - *
  3. add an array of urls as fixed subset (no further crawling done)
    4. - *
- * - * @throws DroidsException - */ - void init() throws DroidsException; - - /** - * Invoke an instance of the worker used in the droid - */ - void start(); - - /** - * Invoke when the droid has completed - */ - void finished(); - - /** - * Return the tasks queue - * - * @return - */ - public Queue getQueue(); - - /** - * Ask the droid for a new worker - */ - Worker getNewWorker(); - - /** - * Get the task master - */ - TaskMaster getTaskMaster(); + + /** + * Invoke an instance of the worker used in the droid + */ + public void start(); + + /** + * Invoke when the droid has completed + */ + public void finished(); + + /** + * Add new Tasks to the queue + * + * @param task the new task + */ + public void add(T task); + + /** + * Ask the droid for a new worker + */ + public Worker getNewWorker(); + + /** + * Parse the task + * + * @param task the task + */ + public void parse(T task) throws DroidsException, IOException; + + /** + * Handle the task. + * + * @param task + */ + public void handle(T task) throws DroidsException, IOException; + + /** + * Filter the task. + * + * @param task + * @return the task or null, if the task is not valid + */ + public T filter(T task); } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Handler.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Handler.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Handler.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Handler.java Fri Dec 7 17:12:35 2012 @@ -35,5 +35,5 @@ public interface Handler { * @throws Exception */ void handle(Task task) - throws IOException, DroidsException; + throws DroidsException, IOException; } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/MultiThreadedTaskMaster.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/MultiThreadedTaskMaster.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/MultiThreadedTaskMaster.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/MultiThreadedTaskMaster.java Fri Dec 7 17:12:35 2012 @@ -101,7 +101,7 @@ public class MultiThreadedTaskMaster queue; private final Worker worker; - public TaskExecutor(Droid droid) + public TaskExecutor(Queue queue, Droid droid) { this.droid = droid; - this.queue = droid.getQueue(); + this.queue = queue; this.worker = droid.getNewWorker(); } Copied: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Parser.java (from r1417006, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/parse/Parser.java) URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Parser.java?p2=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Parser.java&p1=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/parse/Parser.java&r1=1417006&r2=1418399&rev=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/parse/Parser.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Parser.java Fri Dec 7 17:12:35 2012 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.droids.parse; +package org.apache.droids.core; import java.io.IOException; @@ -24,7 +24,6 @@ import org.apache.droids.core.Task; /** * Simple parser that is only forcing to return a parse object. * - * @see Parse * @version 1.0 * */ @@ -32,11 +31,9 @@ public interface Parser { /** * Creates the parse for some content. * - * @param entity - * the underlying stream we are using * @param task * the task that correspond to the stream * @return the parse object */ - Parse parse(Task task) throws DroidsException, IOException; + public void parse(Task task) throws DroidsException, IOException; } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Protocol.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Protocol.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Protocol.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Protocol.java Fri Dec 7 17:12:35 2012 @@ -23,30 +23,28 @@ import java.net.URI; /** * The protocol interface is a wrapper to hide the underlying implementation of * the communication at protocol level. - * + * * @version 1.0 - * */ public interface Protocol { - /** - * Some protocols (like http) offer a mechanism to evaluate whether the client - * can request a given url (in http this is the robots.txt configuration) - * - * @param url - * the url to evaluate - * @return true if we can request the url. false if we are forbidden. - * @throws MalformedURLException - */ - boolean isAllowed(URI url) throws IOException; + /** + * Some protocols (like http) offer a mechanism to evaluate whether the client + * can request a given url (in http this is the robots.txt configuration) + * + * @param url the url to evaluate + * @return true if we can request the url. false if we are forbidden. + * @throws MalformedURLException + */ + boolean isAllowed(URI url) throws IOException; - /** - * Return the content entity represent of the url - * - * @param url - * url of the stream we want to open - * @return the content of the given url - * @throws IOException - */ - Task load(URI uri) throws IOException; + /** + * Return the content entity represent of the url + * + * @param url url of the stream we want to open + * @return the content of the given url + * @throws IOException + */ + Task load(URI uri) throws IOException; + String scheme(); } \ No newline at end of file Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Task.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Task.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Task.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Task.java Fri Dec 7 17:12:35 2012 @@ -16,8 +16,6 @@ */ package org.apache.droids.core; -import java.io.IOException; -import java.io.InputStream; import java.io.Serializable; import java.net.URI; import java.util.Date; @@ -27,42 +25,38 @@ import java.util.Date; * task. That is based on the fact that a droid can extract more tasks from the * one that it is currently working on. However sometimes one want to limit the * number of nested task, this is what is determined by the depth. - * + * * @version 1.0 - * */ public interface Task extends Serializable { - /** - * The id of the task. In a standard crawl that is most likely the url that - * identifies the task - * - * @return The id of the task - */ - public URI getURI(); - - - /** - * The content of the task. - * - * @return the content of the task - * @throws IOException - */ - public InputStream getContent() throws IOException;; - - /** - * - * @return The depth of the task - */ - int getDepth(); - - /** - * When was the task created - * - * @return the date when the task was created. - */ - Date getTaskDate(); - - public void abort(); - - public boolean isAborted(); + /** + * The id of the task. In a standard crawl that is most likely the url that + * identifies the task + * + * @return The id of the task + */ + public URI getURI(); + + /** + * The data of the task. + * + * @return a Map of data values + */ + public ContentEntity getContentEntity(); + + /** + * @return The depth of the task + */ + public int getDepth(); + + /** + * When was the task created + * + * @return the date when the task was created. + */ + public Date getTaskDate(); + + public void abort(); + + public boolean isAborted(); } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Worker.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Worker.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Worker.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/core/Worker.java Fri Dec 7 17:12:35 2012 @@ -19,25 +19,22 @@ package org.apache.droids.core; import java.io.IOException; - - /** * A worker is the unit that is doing the actual work. A {@link Droid} is the * "project manger" that delegates the work to worker units. Worker units are * implemented as threads to scale they number if more work is to do. * * @version 1.0 - * */ public interface Worker { - /** - * Executes a task. - * - * @param task - * @throws DroidsException - * @throws IOException - */ - void execute( final T task ) throws DroidsException, IOException; + /** + * Executes a task. + * + * @param task + * @throws DroidsException + * @throws IOException + */ + public void execute(final T task) throws DroidsException, IOException; } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/AlreadyVisitedFilter.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/AlreadyVisitedFilter.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/AlreadyVisitedFilter.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/AlreadyVisitedFilter.java Fri Dec 7 17:12:35 2012 @@ -37,7 +37,7 @@ public class AlreadyVisitedFilter filter : this.filterChain) { if (filter.filter(task) == null) { return null; Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/Filter.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/Filter.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/Filter.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/Filter.java Fri Dec 7 17:12:35 2012 @@ -33,9 +33,9 @@ public interface Filter * Transforms the URL: can pass the original URL through or "delete" the URL * by returning null * - * @param urlString - * the url to filter + * @param task the task to filter + * * @return null if the filter excludes the url or the url again if allowed */ - Task filter(T task); + public T filter(T task); } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/HostFilter.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/HostFilter.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/HostFilter.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/HostFilter.java Fri Dec 7 17:12:35 2012 @@ -47,7 +47,7 @@ public class HostFilter } @Override - public Task filter(final T task) { + public T filter(final T task) { if (this.allowedHosts.contains(task.getURI().getHost())) { return task; } else { Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/MaxDepthTaskFilter.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/MaxDepthTaskFilter.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/MaxDepthTaskFilter.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/MaxDepthTaskFilter.java Fri Dec 7 17:12:35 2012 @@ -33,7 +33,7 @@ public class MaxDepthTaskFilter 0 && task.getDepth() > maxDepth) { return null; } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/RegexURLFilter.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/RegexURLFilter.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/RegexURLFilter.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/filter/RegexURLFilter.java Fri Dec 7 17:12:35 2012 @@ -67,7 +67,7 @@ public class RegexURLFilter { - - /** - * Run all defined filters. Assume logical AND. - * - * @param task - * the current Task - * @return true if filter plugin accept the url, false if excluded. - */ - public boolean accept(Task task) { - if (task == null) { - return false; - } - - for (String key : getMap().keySet()) { - if (!accept(task, key)) { - return false; - } - } - return true; - } +public class FilterFactory { + private Set filters; - /** - * Run a specific filter class. - * - * @param task - * the task to test - * @param filterName - * - name of the specific filter class. - * @return true if filter plugin accept the url, false if excluded. - */ - public boolean accept(Task task, String filterName) { - if (task == null || doFilter(task, filterName) == null) { - return false; - } - return true; - } + public void addFilter(Filter filter) { + filters.add(filter); + } - /** - * Check string against filters list - * - * @param task - * - task to test - * @param filterName - * - name of the specific filter class. - * @return the URL if it's allowed, NULL otherwise - */ - protected Task doFilter(Task task, String filterName) { - return getMap().get(filterName).filter(task); - } + public T filter(T task) { + for (Filter filter : filters) { + // task = + } + return task; + } } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/HandlerFactory.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/HandlerFactory.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/HandlerFactory.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/HandlerFactory.java Fri Dec 7 17:12:35 2012 @@ -17,6 +17,7 @@ package org.apache.droids.helper.factories; import java.io.IOException; +import java.util.Set; import org.apache.droids.core.DroidsException; import org.apache.droids.core.Handler; @@ -28,26 +29,24 @@ import org.apache.droids.core.Task; * @version 1.0 * */ -public class HandlerFactory extends GenericFactory { +public class HandlerFactory { + private Set handlers; + + public void addHandler(Handler handler) { + handlers.add(handler); + } /** * Will traverse all registered handler and execute them. If we encounter a * problem we directly return false and leave. * - * @param stream - * the underlying stream - * @param url - * the underlying url - * @param parse - * the underlying parse object - * @return false if we found a problem, true if all went well + * @param task the task to handle */ - public boolean handle(Task task) + public void handle(Task task) throws DroidsException, IOException { - for (Handler handler : getMap().values()) { + for (Handler handler : handlers) { handler.handle(task); } - return true; } } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ParserFactory.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ParserFactory.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ParserFactory.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ParserFactory.java Fri Dec 7 17:12:35 2012 @@ -17,30 +17,28 @@ package org.apache.droids.helper.factories; import org.apache.droids.core.DroidsException; -import org.apache.droids.parse.Parser; -import org.apache.droids.parse.ParserNotFoundException; +import org.apache.droids.core.Parser; +import org.apache.droids.core.Task; + +import java.io.IOException; +import java.util.Set; /** * Factory that will lookup a parser by its identifier and return it. - * + * * @version 1.0 - * */ -public class ParserFactory extends GenericFactory { +public class ParserFactory { + private Set parsers; + + public void addParser(Parser parser) { + this.parsers.add(parser); + } - /** - * Lookup a parser by its identifier (content type) and return it. - * - * @param contentType - * for which content type we need a parser - * @return null if we do not find a registered Parser otherwise the Parser - * @throws DroidsException - */ - public Parser getParser(String contentType) throws DroidsException { - if (contentType == null) { - throw new ParserNotFoundException(contentType); + public void parse(Task task) throws DroidsException, IOException { + for (Parser parser : parsers) { + parser.parse(task); + } } - return getMap().get(contentType); - } } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ProtocolFactory.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ProtocolFactory.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ProtocolFactory.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/helper/factories/ProtocolFactory.java Fri Dec 7 17:12:35 2012 @@ -16,39 +16,42 @@ */ package org.apache.droids.helper.factories; -import java.net.URI; - import org.apache.droids.core.Protocol; import org.apache.droids.exception.ProtocolNotFoundException; +import java.net.URI; +import java.util.Set; + /** * Factory that will lookup a protocol plugin and return it. - * + * * @version 1.0 - * */ -public class ProtocolFactory extends GenericFactory { +public class ProtocolFactory { + private Set protocols; + + public void addProtocol(Protocol protocol) { + protocols.add(protocol); + } + - /** - * Will lookup a protocol based on the underlying uri - * - * @param uri - * the string that contains the protocol - * @return ready to use protocol plugin or null if non have been found - * @throws ProtocolNotFoundException - */ - public Protocol getProtocol(URI uri) throws ProtocolNotFoundException { - Protocol protocol = null; - try { - String protocolName = uri.getScheme(); - if (protocolName == null) { + /** + * Will lookup a protocol based on the underlying uri + * + * @param uri the string that contains the protocol + * @return ready to use protocol plugin or null if non have been found + * @throws ProtocolNotFoundException + */ + public Protocol getProtocol(URI uri) throws ProtocolNotFoundException { + String scheme = uri.getScheme(); + if (scheme == null) { + throw new ProtocolNotFoundException(uri); + } + for (Protocol protocol : protocols) { + if (protocol.scheme().equals(scheme)) + return protocol; + } throw new ProtocolNotFoundException(uri); - } - protocol = getMap().get(protocolName); - } catch (ProtocolNotFoundException e) { - throw new ProtocolNotFoundException(uri, e.toString()); } - return protocol; - } } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/SimpleTask.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/SimpleTask.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/SimpleTask.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/SimpleTask.java Fri Dec 7 17:12:35 2012 @@ -9,6 +9,7 @@ public class SimpleTask implements Task private URI uri; private int depth; private boolean aborted; + private ContentEntity contentEntity; private static final long serialVersionUID = 2506491803180939447L; @@ -16,19 +17,20 @@ public class SimpleTask implements Task this.uri = uri; this.depth = depth; this.aborted = false; + this.contentEntity = new ContentEntity(); } - - @Override + + @Override + public ContentEntity getContentEntity() { + return contentEntity; + } + + @Override public URI getURI() { return uri; } @Override - public InputStream getContent() throws IOException { - return null; - } - - @Override public int getDepth() { return depth; } Modified: incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/TestSimpleQueue.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/TestSimpleQueue.java?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/TestSimpleQueue.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-core/src/test/java/org/apache/droids/core/TestSimpleQueue.java Fri Dec 7 17:12:35 2012 @@ -16,40 +16,39 @@ */ package org.apache.droids.core; -import java.io.File; +import java.net.URI; import junit.framework.Assert; import org.apache.droids.filter.MaxDepthTaskFilter; -import org.apache.droids.robot.walker.FileTask; import org.junit.Before; import org.junit.Test; public class TestSimpleQueue { - MaxDepthTaskFilter filter; + MaxDepthTaskFilter filter; @Before public final void initialize() { - filter = new MaxDepthTaskFilter(); + filter = new MaxDepthTaskFilter(); filter.setMaxDepth(5); } @Test public void whenTaskBelowMaxDepthIsValidated_thenTaskIsValid() throws Exception { - final FileTask task = new FileTask(new File(""), 3); + final SimpleTask task = new SimpleTask(new URI("http://www.example.com"), 3); Assert.assertNotNull(filter.filter(task)); } @Test public void whenTaskEqualToMaxDepthIsValidated_thenTaskIsValid() throws Exception { - final FileTask task = new FileTask(new File(""), 5); + final SimpleTask task = new SimpleTask(new URI("http://www.example.com"), 5); Assert.assertNotNull(filter.filter(task)); } @Test public void whenTaskOverMaxDepthIsValidated_thenTaskIsNotValid() throws Exception { - final FileTask task = new FileTask(new File(""), 7); + final SimpleTask task = new SimpleTask(new URI("http://www.example.com"), 7); Assert.assertNull(filter.filter(task)); } Modified: incubator/droids/branches/0.2.x-cleanup/droids-crawler/pom.xml URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/pom.xml?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-crawler/pom.xml (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/pom.xml Fri Dec 7 17:12:35 2012 @@ -7,16 +7,15 @@ droids 0.3.0-incubating-SNAPSHOT - org.apache.droids droids-crawler - 0.3.0-incubating-SNAPSHOT droids-crawler - http://maven.apache.org - - UTF-8 - + org.apache.droids + droids-core + ${project.version} + + junit junit 3.8.1 Copied: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java (from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingDroid.java) URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java?p2=incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java&p1=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingDroid.java&r1=1406628&r2=1418399&rev=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingDroid.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java Fri Dec 7 17:12:35 2012 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.droids.robot.crawler; +package org.apache.droids.crawler; import java.net.URI; import java.net.URISyntaxException; @@ -23,103 +23,49 @@ import java.util.Collection; import com.google.common.base.Preconditions; import java.util.Queue; -import org.apache.droids.AbstractDroid; -import org.apache.droids.LinkTask; -import org.apache.droids.api.Link; -import org.apache.droids.api.TaskMaster; -import org.apache.droids.api.TaskValidator; -import org.apache.droids.api.Worker; +import org.apache.droids.core.AbstractDroid; +import org.apache.droids.core.TaskMaster; +import org.apache.droids.core.Worker; import org.apache.droids.exception.InvalidTaskException; -import org.apache.droids.helper.factories.ParserFactory; -import org.apache.droids.helper.factories.ProtocolFactory; -import org.apache.droids.helper.factories.URLFiltersFactory; - -public abstract class CrawlingDroid extends AbstractDroid -{ - - private Collection initialLocations; - ProtocolFactory protocolFactory; - ParserFactory parserFactory; - URLFiltersFactory filtersFactory; - private TaskValidator linkValidator; - - public CrawlingDroid(Queue queue, TaskMaster taskMaster) - { - super(queue, taskMaster); - } - - public void setInitialLocations(Collection initialLocations) - { - this.initialLocations = initialLocations; - } - - @Override - public void init() throws InvalidTaskException - { - Preconditions.checkState(initialLocations != null, "WebCrawlerDroid requires at least one starting file"); - Preconditions.checkState(!initialLocations.isEmpty(), "WebCrawlerDroid requires at least one starting file"); - for (String location : initialLocations) { - URI uri; - try { - uri = new URI(location); - } catch (URISyntaxException ex) { - throw new InvalidTaskException("Invalid lication: " + location); - } - queue.offer(new LinkTask(null, uri, 0)); - } - } - - public void start() - { - taskMaster.start(queue, this); - } - - @Override - public void finished() - { - log.info("FINISHED!!!"); - } - - public abstract Worker getNewWorker(); - - public ProtocolFactory getProtocolFactory() - { - return protocolFactory; - } - - public void setProtocolFactory(ProtocolFactory protocolFactory) - { - this.protocolFactory = protocolFactory; - } - - public ParserFactory getParserFactory() - { - return parserFactory; - } - - public void setParserFactory(ParserFactory parserFactory) - { - this.parserFactory = parserFactory; - } - - public URLFiltersFactory getFiltersFactory() - { - return filtersFactory; - } - - public void setFiltersFactory(URLFiltersFactory filtersFactory) - { - this.filtersFactory = filtersFactory; - } - - public void setLinkValidator(TaskValidator linkValidator) - { - this.linkValidator = linkValidator; - } - - public TaskValidator getLinkValidator() - { - return linkValidator; - } - + +public abstract class CrawlingDroid extends AbstractDroid { + + private Collection initialLocations; + + public CrawlingDroid(Queue queue, TaskMaster taskMaster) { + super(queue, taskMaster); + } + + public void setInitialLocations(Collection initialLocations) { + this.initialLocations = initialLocations; + } + + @Override + public void init() throws InvalidTaskException { + Preconditions.checkState(initialLocations != null, + "WebCrawlerDroid requires at least one starting file"); + Preconditions.checkState(!initialLocations.isEmpty(), + "WebCrawlerDroid requires at least one starting file"); + for (String location : initialLocations) { + URI uri; + try { + uri = new URI(location); + } catch (URISyntaxException ex) { + throw new InvalidTaskException("Invalid lication: " + location); + } + queue.offer(new LinkTask(null, uri, 0)); + } + } + + public void start() { + taskMaster.start(queue, this); + } + + @Override + public void finished() { + logger.info("FINISHED!!!"); + } + + public abstract Worker getNewWorker(); + } Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingDroid.java ------------------------------------------------------------------------------ svn:mergeinfo = Copied: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingWorker.java (from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingWorker.java) URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingWorker.java?p2=incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingWorker.java&p1=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingWorker.java&r1=1406628&r2=1418399&rev=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/CrawlingWorker.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingWorker.java Fri Dec 7 17:12:35 2012 @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.droids.robot.crawler; +package org.apache.droids.crawler; import java.io.IOException; import java.net.URI; @@ -22,136 +22,117 @@ import java.util.Collection; import java.util.LinkedHashMap; import java.util.Map; -import org.apache.droids.api.ContentEntity; -import org.apache.droids.api.Link; -import org.apache.droids.api.ManagedContentEntity; -import org.apache.droids.api.Parse; -import org.apache.droids.api.Parser; -import org.apache.droids.api.Protocol; -import org.apache.droids.api.Task; -import org.apache.droids.api.TaskValidator; -import org.apache.droids.api.Worker; -import org.apache.droids.exception.DroidsException; +import org.apache.droids.core.DroidsException; +import org.apache.droids.core.Protocol; +import org.apache.droids.core.Task; +import org.apache.droids.core.Worker; import org.apache.droids.helper.factories.HandlerFactory; import org.apache.droids.helper.factories.URLFiltersFactory; +import org.apache.droids.parse.Parse; +import org.apache.droids.core.Parser; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -public class CrawlingWorker implements Worker -{ +public class CrawlingWorker implements Worker { - private static final Logger LOG = LoggerFactory.getLogger(CrawlingWorker.class); + private static final Logger LOG = LoggerFactory.getLogger(CrawlingWorker.class); - private final CrawlingDroid droid; - HandlerFactory handlerFactory; - - public CrawlingWorker( CrawlingDroid droid ) - { - this.droid = droid; - } - - @Override - public void execute(Link link) throws DroidsException, IOException - { - final String userAgent = this.getClass().getCanonicalName(); - if (LOG.isDebugEnabled()) { - LOG.debug("Starting " + userAgent); - } - URI uri = link.getURI(); - final Protocol protocol = droid.getProtocolFactory().getProtocol(uri); - if (protocol == null) { - if (LOG.isWarnEnabled()) { - LOG.warn("Unsupported protocol scheme '" + uri.getScheme() + "'"); - } - return; - } - - if (protocol.isAllowed(uri)) { - if (LOG.isInfoEnabled()) { - LOG.info("Loading " + uri); - } - ManagedContentEntity entity = null; - try { - entity = protocol.load(uri); - } catch(OutOfMemoryError e) { - LOG.error("Out of memory processing: " + uri + " skipping", e); - throw new DroidsException(e); - } - try { - String contentType = entity.getMimeType(); - if (LOG.isDebugEnabled()) { - LOG.debug("Content type " + contentType); - } - if (contentType == null){ - LOG.info("Missing content type... can't parse..."); - } - else { - Parser parser = droid.getParserFactory().getParser(contentType); - if( parser == null ) { - if (LOG.isDebugEnabled()) { - LOG.debug("Could not find parser for " + contentType); - } - } - else { - Parse parse = parser.parse(entity, link); - if( parse.getNewTasks() != null && parse.isFollowed() ) { - Collection outlinks = getFilteredOutlinks( parse ); - droid.getQueue().addAll( outlinks ); - } - entity.setParse(parse); - handle(entity, link); - } - } - } finally { - entity.finish(); - } - } - else { - if (LOG.isInfoEnabled()) { - LOG.info("Stopping processing since" - + " bots are not allowed for " + uri ); - } - } - } - - protected void handle(ContentEntity entity, Link link) - throws DroidsException, IOException - { - getHandlerFactory().handle(link.getURI(), entity); - } - - protected Collection getFilteredOutlinks( Parse parse ) - { - URLFiltersFactory filters = droid.getFiltersFactory(); - TaskValidator< Link > linkValidator = droid.getLinkValidator(); - - // TODO -- make the hashvalue for Outlink... - Map filtered = new LinkedHashMap(); - for( Task outTask : parse.getNewTasks() ) { - // only use Links, so if for some reason it isn't a Link, skip - if( !(outTask instanceof Link)) { - continue; - } - Link outlink = (Link)outTask; - String id = outlink.getId(); - if (filters.accept(id) && !filtered.containsKey(id)) { - if( linkValidator == null ){ - filtered.put(id,outlink); - } - else if( linkValidator.validate( outlink ) ){ - filtered.put(id,outlink); - } - } - } - return filtered.values(); - } - - public HandlerFactory getHandlerFactory() { - return handlerFactory; - } - - public void setHandlerFactory(HandlerFactory handlerFactory) { - this.handlerFactory = handlerFactory; - } -} + private final CrawlingDroid droid; + HandlerFactory handlerFactory; + public CrawlingWorker(CrawlingDroid droid) { + this.droid = droid; + } + + @Override + public void execute(Link link) throws DroidsException, IOException { + final String userAgent = this.getClass().getCanonicalName(); + if (LOG.isDebugEnabled()) { + LOG.debug("Starting " + userAgent); + } + URI uri = link.getURI(); + final Protocol protocol = droid.getProtocolFactory().getProtocol(uri); + if (protocol == null) { + if (LOG.isWarnEnabled()) { + LOG.warn("Unsupported protocol scheme '" + uri.getScheme() + "'"); + } + return; + } + + if (protocol.isAllowed(uri)) { + if (LOG.isInfoEnabled()) { + LOG.info("Loading " + uri); + } +// ContentEntity entity = null; + try { +// entity = protocol.load(uri); + } catch (OutOfMemoryError e) { + LOG.error("Out of memory processing: " + uri + " skipping", e); + throw new DroidsException(e); + } + try { +// String contentType = entity.getMimeType(); + String contentType = ""; + if (LOG.isDebugEnabled()) { + LOG.debug("Content type " + contentType); + } + if (contentType == null) { + LOG.info("Missing content type... can't parse..."); + } else { + Parser parser = droid.getParserFactory().getParser(contentType); + if (parser == null) { + if (LOG.isDebugEnabled()) { + LOG.debug("Could not find parser for " + contentType); + } + } else { +// Parse parse = parser.parse(entity, link); + Parse parse = null; + if (parse.getNewTasks() != null && parse.isFollowed()) { + Collection outlinks = getFilteredOutlinks(parse); + droid.getQueue().addAll(outlinks); + } +// entity.setParse(parse); + handle(link); + } + } + } finally { +// entity.finish(); + } + } else { + if (LOG.isInfoEnabled()) { + LOG.info("Stopping processing since" + " bots are not allowed for " + uri); + } + } + } + + protected void handle(Task task) throws DroidsException, IOException { + getHandlerFactory().handle(task); + } + + protected Collection getFilteredOutlinks(Parse parse) { + URLFiltersFactory filters = droid.getFiltersFactory(); + + // TODO -- make the hashvalue for Outlink... + Map filtered = new LinkedHashMap(); + for (Task outTask : parse.getNewTasks()) { + // only use Links, so if for some reason it isn't a Link, skip + if (!(outTask instanceof Link)) { + continue; + } + Link outlink = (Link) outTask; + URI uri = outlink.getURI(); + if (filters.accept(outlink) && !filtered.containsKey(uri)) { + filtered.put(uri, outlink); + } + } + return filtered.values(); + } + + public HandlerFactory getHandlerFactory() { + return handlerFactory; + } + + public void setHandlerFactory(HandlerFactory handlerFactory) { + this.handlerFactory = handlerFactory; + } +} Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/CrawlingWorker.java ------------------------------------------------------------------------------ svn:eol-style = native Copied: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/Link.java (from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/api/Link.java) URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/Link.java?p2=incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/Link.java&p1=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/api/Link.java&r1=1406628&r2=1418399&rev=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/api/Link.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/Link.java Fri Dec 7 17:12:35 2012 @@ -14,12 +14,14 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.droids.api; +package org.apache.droids.crawler; import java.net.URI; import java.util.Collection; import java.util.Date; +import org.apache.droids.core.Task; + /** * Simple extension of a {@link Task}. Adding from/to link, anchor text @@ -30,11 +32,6 @@ import java.util.Date; */ public interface Link extends Task { /** - * @return the URI to this link - */ - URI getURI(); - - /** * @return the Anchor text for this link */ String getAnchorText(); Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/Link.java ------------------------------------------------------------------------------ svn:eol-style = native Copied: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/LinkTask.java (from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/LinkTask.java) URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/LinkTask.java?p2=incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/LinkTask.java&p1=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/LinkTask.java&r1=1406628&r2=1418399&rev=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/LinkTask.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/LinkTask.java Fri Dec 7 17:12:35 2012 @@ -14,180 +14,182 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.droids; +package org.apache.droids.crawler; import java.net.URI; import java.util.Collection; import java.util.Date; +import java.io.IOException; +import java.io.InputStream; import java.io.Serializable; -import org.apache.droids.api.Link; /** * - * Basic implementation for @Link. - * LinkTasks are working instructions for URI based droids. + * Basic implementation for @Link. LinkTasks are working instructions for URI + * based droids. * */ public class LinkTask implements Link, Serializable { - private static final long serialVersionUID = -44808094386453088L; + private static final long serialVersionUID = -44808094386453088L; - private Date started; - private final int depth; - private final URI uri; - private final Link from; - - private Date lastModifedDate; - private Collection linksTo; - private String anchorText; - private int weight; - private boolean aborted = false; - - /** - * Creates a new LinkTask. - * - * @param from - * @param uri - * @param depth - */ - public LinkTask(Link from, URI uri, int depth) { - this.from = from; - this.uri = uri; - this.depth = depth; - this.started = new Date(); - } - - /** - * Creates a new LinkTask. - * - * @param from - * @param uri - * @param depth - * @param weight - */ - public LinkTask(Link from, URI uri, int depth, int weight) { - this.from = from; - this.uri = uri; - this.depth = depth; - this.started = new Date(); - this.weight = weight; - } - - /** - * Creates a new LinkTask - * - * @param from - * @param uri - * @param depth - * @param anchorText - */ - public LinkTask(Link from, URI uri, int depth, String anchorText) { - this(from, uri, depth); - this.anchorText = anchorText; - } - - @Override - public String getId() { - return uri.toString(); - } - - @Override - public Date getTaskDate() { - return started; - } - - /** - * Set the Date the task started. - * - * @param started - */ - public void setTaskDate(Date started) { - this.started = started; - } - - @Override - public int getDepth() { - return depth; - } - - @Override - public Link getFrom() { - return from; - } - - @Override - public Collection getTo() { - return linksTo; - } - - @Override - public Date getLastModifiedDate() { - return lastModifedDate; - } - - /** - * Set the Date the Task object was last modified. - * - * @param lastModifedDate - */ - public void setLastModifedDate(Date lastModifedDate) { - this.lastModifedDate = lastModifedDate; - } - - /** - * Set Outgoing links. - * - * @param linksTo - */ - public void setLinksTo(Collection linksTo) { - this.linksTo = linksTo; - } - - @Override - public URI getURI() { - return uri; - } - - @Override - public String getAnchorText() { - return anchorText; - } - - /** - * Set the anchor text for this link. - * - * @param anchorText - */ - public void setAnchorText(String anchorText) { - this.anchorText = anchorText; - } - - /** - * Get the weight of the link - * - * @return the links weight - */ - public int getWeight() { - return weight; - } - - /** - * Set the weight of the link. - * - * @param weight - */ - public void setWeight(int weight) { - this.weight = weight; - } - - @Override - public void abort() { - aborted = true; - } - - @Override - public boolean isAborted() { - return aborted; - } + private Date started; + private final int depth; + private final URI uri; + private final Link from; + + private Date lastModifiedDate; + private Collection linksTo; + private String anchorText; + private int weight; + private boolean aborted = false; + + /** + * Creates a new LinkTask. + * + * @param from + * @param uri + * @param depth + */ + public LinkTask(Link from, URI uri, int depth) { + this.from = from; + this.uri = uri; + this.depth = depth; + this.started = new Date(); + } + + /** + * Creates a new LinkTask. + * + * @param from + * @param uri + * @param depth + * @param weight + */ + public LinkTask(Link from, URI uri, int depth, int weight) { + this.from = from; + this.uri = uri; + this.depth = depth; + this.started = new Date(); + this.weight = weight; + } + + /** + * Creates a new LinkTask + * + * @param from + * @param uri + * @param depth + * @param anchorText + */ + public LinkTask(Link from, URI uri, int depth, String anchorText) { + this(from, uri, depth); + this.anchorText = anchorText; + } + + @Override + public URI getURI() { + return uri; + } + + @Override + public InputStream getContent() throws IOException { + // TODO Auto-generated method stub + return null; + } + + @Override + public Date getTaskDate() { + return started; + } + + /** + * Set the Date the task started. + * + * @param started + */ + public void setTaskDate(Date started) { + this.started = started; + } + + @Override + public int getDepth() { + return depth; + } + + @Override + public Link getFrom() { + return from; + } + + @Override + public Collection getTo() { + return linksTo; + } + + @Override + public Date getLastModifiedDate() { + return lastModifiedDate; + } + + /** + * Set the Date the Task object was last modified. + * + * @param lastModifiedDate + */ + public void setLastModifiedDate(Date lastModifiedDate) { + this.lastModifiedDate = lastModifiedDate; + } + + /** + * Set Outgoing links. + * + * @param linksTo + */ + public void setLinksTo(Collection linksTo) { + this.linksTo = linksTo; + } + + @Override + public String getAnchorText() { + return anchorText; + } + + /** + * Set the anchor text for this link. + * + * @param anchorText + */ + public void setAnchorText(String anchorText) { + this.anchorText = anchorText; + } + + /** + * Get the weight of the link + * + * @return the links weight + */ + public int getWeight() { + return weight; + } + + /** + * Set the weight of the link. + * + * @param weight + */ + public void setWeight(int weight) { + this.weight = weight; + } + + @Override + public void abort() { + aborted = true; + } + + @Override + public boolean isAborted() { + return aborted; + } } \ No newline at end of file Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/LinkTask.java ------------------------------------------------------------------------------ svn:eol-style = native Copied: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java (from r1406628, incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/ReportCrawlingDroid.java) URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java?p2=incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java&p1=incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/ReportCrawlingDroid.java&r1=1406628&r2=1418399&rev=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-core/src/main/java/org/apache/droids/robot/crawler/ReportCrawlingDroid.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java Fri Dec 7 17:12:35 2012 @@ -16,12 +16,11 @@ * specific language governing permissions and limitations * under the License. */ -package org.apache.droids.robot.crawler; +package org.apache.droids.crawler; import java.util.Queue; -import org.apache.droids.api.Link; -import org.apache.droids.api.TaskMaster; -import org.apache.droids.api.Worker; +import org.apache.droids.core.TaskMaster; +import org.apache.droids.core.Worker; import org.apache.droids.handle.ReportHandler; import org.apache.droids.helper.factories.HandlerFactory; Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java ------------------------------------------------------------------------------ svn:keywords = Author Date Id Revision Propchange: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/crawler/ReportCrawlingDroid.java ------------------------------------------------------------------------------ svn:mime-type = text/plain Modified: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpContentEntity.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpContentEntity.java?rev=1418399&r1=1406628&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpContentEntity.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpContentEntity.java Fri Dec 7 17:12:35 2012 @@ -23,7 +23,7 @@ import java.util.HashMap; import java.util.Map; import java.util.Set; -import org.apache.droids.api.AdvancedManagedContentEntity; +import org.apache.droids.core.AdvancedManagedContentEntity; import org.apache.http.Header; import org.apache.http.HttpEntity; Modified: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpProtocol.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpProtocol.java?rev=1418399&r1=1406628&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpProtocol.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/AdvancedHttpProtocol.java Fri Dec 7 17:12:35 2012 @@ -21,7 +21,7 @@ package org.apache.droids.protocol.http; import java.io.IOException; import java.net.URI; -import org.apache.droids.api.AdvancedManagedContentEntity; +import org.apache.droids.core.AdvancedManagedContentEntity; import org.apache.http.HttpEntity; import org.apache.http.HttpResponse; import org.apache.http.HttpStatus; Modified: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpContentEntity.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpContentEntity.java?rev=1418399&r1=1406628&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpContentEntity.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpContentEntity.java Fri Dec 7 17:12:35 2012 @@ -20,8 +20,8 @@ import java.io.IOException; import java.io.InputStream; import java.util.Locale; -import org.apache.droids.api.ManagedContentEntity; -import org.apache.droids.api.Parse; +import org.apache.droids.core.ManagedContentEntity; +import org.apache.droids.parse.Parse; import org.apache.http.Header; import org.apache.http.HeaderElement; import org.apache.http.HttpEntity; Modified: incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpProtocol.java URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpProtocol.java?rev=1418399&r1=1406628&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpProtocol.java (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-crawler/src/main/java/org/apache/droids/protocol/http/HttpProtocol.java Fri Dec 7 17:12:35 2012 @@ -20,8 +20,8 @@ import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; -import org.apache.droids.api.ManagedContentEntity; -import org.apache.droids.api.Protocol; +import org.apache.droids.core.ManagedContentEntity; +import org.apache.droids.core.Protocol; import org.apache.droids.norobots.ContentLoader; import org.apache.droids.norobots.NoRobotClient; import org.apache.droids.norobots.NoRobotException; Modified: incubator/droids/branches/0.2.x-cleanup/droids-spring/src/main/resources/org/apache/droids/dynamic/droids-core-context.xml URL: http://svn.apache.org/viewvc/incubator/droids/branches/0.2.x-cleanup/droids-spring/src/main/resources/org/apache/droids/dynamic/droids-core-context.xml?rev=1418399&r1=1418398&r2=1418399&view=diff ============================================================================== --- incubator/droids/branches/0.2.x-cleanup/droids-spring/src/main/resources/org/apache/droids/dynamic/droids-core-context.xml (original) +++ incubator/droids/branches/0.2.x-cleanup/droids-spring/src/main/resources/org/apache/droids/dynamic/droids-core-context.xml Fri Dec 7 17:12:35 2012 @@ -58,7 +58,7 @@ - + + ref="org.apache.droids.helper.factories.FilterFactory"/> @@ -62,8 +62,8 @@ - + - +