I'm working on a solution for this problem and I think that the best fix would be to move the HandlerFactory object from CrawlingDroid to CrawlingWorker. With the cloneable solution, every handler must override the clone() method and implements the interface Cloneable. I think that moving the factory to the worker could be the best solution, because with each thread (worker) you will have a new factory object with new handlers and you we can to forget about the clone implementation. Also, the clone solution will create and destroy all handlers for each target handle in each thread. Opposite, the move fix will take more memory but all the threads will have it's own handlers. I have a problem about this fix, I wonder where can be the HandlerFactory init for the Worker. What do you think about? Salu2.
Hello, For those living in or near NYC, you may be interested in joining (and/or presenting?) at the NYC Search & Discovery Meetup. Topics are: search, machine learning, data mining, NLP, information gathering, information extraction, etc. http://www.meetup.com/NYC-Search-and-Discovery/ Our previous/first meetup was about solr-python and parse.ly (a service that makes use of Solr and solr-python). Tomorrow (December 2 2009) we have: Incorporating Probabilistic Retrieval Knowledge into TFIDF-based Search Engine You can RSVP at: http://www.meetup.com/NYC-Search-and-Discovery/calendar/11745435/ Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
[ https://issues.apache.org/jira/browse/DROIDS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorsten Scherler closed DROIDS-69.
-----------------------------------
Resolution: Fixed
http://incubator.apache.org/droids/api/
committed revision 880695
> Add link to Javadoc
> -------------------
>
> Key: DROIDS-69
> URL: https://issues.apache.org/jira/browse/DROIDS-69
> Project: Droids
> Issue Type: Improvement
> Components: documentation
> Reporter: Otis Gospodnetic
> Priority: Minor
>
> It would be good to add a link to Droids Javadoc (trunk and the latest release when there
is one).
> http://ci.apache.org/projects/droids/api/overview-summary.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Personally I used Droids to crawl a website of approximately 250000 pages. The queue was stored in memory and I arbitrarily allocated 1GB of memory to java. Everything worked fine. That's not a large number of webpages but I think droids' current implementation is well suited for such jobs: crawling a relatively small set of webpage or crawling an intranet. This is particularly right if you need to customize the handling process of the pages. I Hope this experience may help. Bertil Chapuis On Nov 14, 2009, at 3:59 AM, Otis Gospodnetic wrote: > OK, thanks. > > So how do people really use Droids at scale? e.g. crawling a large number of web pages? I happen to use it for something smalish, so I never had issues with the queue being in the JVM heap and getting OOMs because of that. But I imagine that anyone using it for a larger crawl would hit OOM sooner or later, no? > > Does this imply that either nobody is using Droids for large-scale crawls, or that everyone who does implemented their own, custom disk-backed queue? > > > Thanks, > Otis > -- > Sematext is hiring -- http://sematext.com/about/jobs.html?mls > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR > > > > ----- Original Message ---- >> From: Ryan McKinley <ryantxu@gmail.com> >> To: droids-dev@incubator.apache.org >> Sent: Fri, November 13, 2009 5:17:51 PM >> Subject: Re: Queue: in memory or on disk? >> >> ya, the standard one is in memory. >> >> It is easy to write one to store things to disk or whatever -- I use one that >> stores tasks to an h2 database, but it is not general enough to contribute >> back... >> >> I think Migfa was looking at replacing the droids Queue interface with a >> standard java.util.Queue interface >> >> ryan >> >> >> On Nov 13, 2009, at 5:10 PM, Chapuis Bertil wrote: >> >>> I think the current implementation only provides in memory queues of tasks. >> However, since the TaskQueue interface is relatively simple it shouldn't be too >> hard to persists the data on the disk or to implement a TaskQueue which works >> with a JMS broker or something else. >>> >>> >>> On Nov 12, 2009, at 10:37 PM, Otis Gospodnetic wrote: >>> >>>> Hello, >>>> >>>> I haven't looked at the sources. But who stores items put in the Queue? Are >> they in memory, or does something write them to disk, or something else? >>>> >>>> Thanks, >>>> Otis >>>> -- >>>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls >>>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR >>>> >>> >
Add link to Javadoc
-------------------
Key: DROIDS-69
URL: https://issues.apache.org/jira/browse/DROIDS-69
Project: Droids
Issue Type: Improvement
Components: documentation
Reporter: Otis Gospodnetic
Priority: Minor
It would be good to add a link to Droids Javadoc (trunk and the latest release when there
is one).
http://ci.apache.org/projects/droids/api/overview-summary.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
OK, thanks. So how do people really use Droids at scale? e.g. crawling a large number of web pages? I happen to use it for something smalish, so I never had issues with the queue being in the JVM heap and getting OOMs because of that. But I imagine that anyone using it for a larger crawl would hit OOM sooner or later, no? Does this imply that either nobody is using Droids for large-scale crawls, or that everyone who does implemented their own, custom disk-backed queue? Thanks, Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR ----- Original Message ---- > From: Ryan McKinley <ryantxu@gmail.com> > To: droids-dev@incubator.apache.org > Sent: Fri, November 13, 2009 5:17:51 PM > Subject: Re: Queue: in memory or on disk? > > ya, the standard one is in memory. > > It is easy to write one to store things to disk or whatever -- I use one that > stores tasks to an h2 database, but it is not general enough to contribute > back... > > I think Migfa was looking at replacing the droids Queue interface with a > standard java.util.Queue interface > > ryan > > > On Nov 13, 2009, at 5:10 PM, Chapuis Bertil wrote: > > > I think the current implementation only provides in memory queues of tasks. > However, since the TaskQueue interface is relatively simple it shouldn't be too > hard to persists the data on the disk or to implement a TaskQueue which works > with a JMS broker or something else. > > > > > > On Nov 12, 2009, at 10:37 PM, Otis Gospodnetic wrote: > > > >> Hello, > >> > >> I haven't looked at the sources. But who stores items put in the Queue? Are > they in memory, or does something write them to disk, or something else? > >> > >> Thanks, > >> Otis > >> -- > >> Sematext is hiring -- http://sematext.com/about/jobs.html?mls > >> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR > >> > >
ya, the standard one is in memory. It is easy to write one to store things to disk or whatever -- I use one that stores tasks to an h2 database, but it is not general enough to contribute back... I think Migfa was looking at replacing the droids Queue interface with a standard java.util.Queue interface ryan On Nov 13, 2009, at 5:10 PM, Chapuis Bertil wrote: > I think the current implementation only provides in memory queues of > tasks. However, since the TaskQueue interface is relatively simple > it shouldn't be too hard to persists the data on the disk or to > implement a TaskQueue which works with a JMS broker or something else. > > > On Nov 12, 2009, at 10:37 PM, Otis Gospodnetic wrote: > >> Hello, >> >> I haven't looked at the sources. But who stores items put in the >> Queue? Are they in memory, or does something write them to disk, >> or something else? >> >> Thanks, >> Otis >> -- >> Sematext is hiring -- http://sematext.com/about/jobs.html?mls >> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR >> >
I think the current implementation only provides in memory queues of tasks. However, since the TaskQueue interface is relatively simple it shouldn't be too hard to persists the data on the disk or to implement a TaskQueue which works with a JMS broker or something else. On Nov 12, 2009, at 10:37 PM, Otis Gospodnetic wrote: > Hello, > > I haven't looked at the sources. But who stores items put in the Queue? Are they in memory, or does something write them to disk, or something else? > > Thanks, > Otis > -- > Sematext is hiring -- http://sematext.com/about/jobs.html?mls > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR >
Hello, Here is the java doc. http://ci.apache.org/projects/droids/api/overview-summary.html Best regards, Bertil On Nov 12, 2009, at 10:38 PM, Otis Gospodnetic wrote: > vadoc, hoping my question about Queue would be in the javadoc, but
Hi, I tried looking for Droids javadoc, hoping my question about Queue would be in the javadoc, but I could not find the javadoc anywhere on Droids' site. I then tried looking for the link to the svn web view (so I don't have to svn co), but I couldn't find that either. Am I just not seeing them? Thanks, Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
Hello, I haven't looked at the sources. But who stores items put in the Queue? Are they in memory, or does something write them to disk, or something else? Thanks, Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
2009/11/12 Chapuis Bertil <bchapuis@agimem.com> > Not in a short scope, I am quite busy this month. But Javier could create a > ticket for the issue. > > Best regards, > > Bertil Chapuis > Ticket created on https://issues.apache.org/jira/browse/DROIDS-68 I add the synchroniced fix as a workaround in the ticket for a fast solution. The cloneable fix is better but I'm busy too. Best regards.
HandlerFactory fails with multithreaded implementation
------------------------------------------------------
Key: DROIDS-68
URL: https://issues.apache.org/jira/browse/DROIDS-68
Project: Droids
Issue Type: Bug
Components: core
Environment: Ubuntu 9.04 i386
Java Runtime 1.6_16
Reporter: Javier Puerto
Priority: Critical
Hi, I'm working with Droids and made some URL crawlers to save a lot of web pages in disk.
In JUnit test, I run a little http server and crawl 20 pages, the most times everything works
ok but in rare cases I get an error. I found the problem in the HandlerFactory implementation,
in the example the call to handlers is like this:
protected void handle(ContentEntity entity, Link link)
throws DroidsException, IOException
{
droid.getHandlerFactory().handle(link.getURI(), entity);
}
If two or more workers is trying to handle at same time, the HandlerFactory will handle with
the same instance. The solution could be saving memory or improving performance.
The first solution could be implemented adding a "synchronized" to HandlerFactory.handle like
this.
public synchronized boolean handle(URI uri, ContentEntity entity)
throws DroidsException, IOException {
for (Handler handler : getMap().values()) {
handler.handle(uri, entity);
}
return true;
}
This solution works but it is a workaround.
The real solution was discussed in the dev list and it was make the Droid and the GenericFactory
abstractions clonable and invoking the clone method in the Worker's constructor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
On Wed, Nov 11, 2009 at 9:25 AM, Thorsten Scherler < thorsten.scherler.ext@juntadeandalucia.es> wrote: > On Tue, 2009-11-10 at 08:40 +0100, Chapuis Bertil wrote: > > I had the same problem and solved it in my handler's implementation by > only > > using local variables and limited concurrent accesses. As I understand > the > > issue the same limitation could occurs with a custom CrawlingDroid > > implementation since all workers are using the same Droid. A nice fix > could > > be to make the Droid and the GenericFactory abstractions clonable and > > invoking the clone method in the Worker's constructor. > > Could you provide a patch? Not in a short scope, I am quite busy this month. But Javier could create a ticket for the issue. Best regards, Bertil Chapuis
On Tue, 2009-11-03 at 09:53 +0100, Thorsten Scherler wrote: > On Tue, 2009-11-03 at 09:34 +0100, Thorsten Scherler wrote: > > The droids pmc proposes Javier Purto to become a new committer. During the time period there were no negative votes, and more than 3 positive votes. So Javier welcome as a new Apache Droids committer! Here are the next steps. There is no rush. You can provide the answers here or use the droids-private AT incubator.a.o address if you prefer. We are generally following the procedure at: http://www.apache.org/dev/#pmc http://www.apache.org/dev/pmc.html#newcommitter You need to send a Contributor License Agreement to the ASF. Normally you would send just an Individual CLA. If you also make contributions done in work time or using work resources then see the additional Corporate CLA. It is up to you if you need that additional CLA, as the Individual CLA declares that you are legally entitled. Ask us if you have any issues. http://www.apache.org/licenses/#clas You need to choose a preferred ASF user name and alternatives. See the existing names http://www.apache.org/~jim/committers.html When we see the ASF volunteer secretary record the receipt of the CLA in an svn commit, we can proceed to ask Infrastructure to set up your account. The developer section of the website describes the roles and provides other resources. Especially important are the ones for "new committers". http://www.apache.org/foundation/how-it-works.html http://www.apache.org/dev/ salu2 -- Thorsten Scherler <thorsten.at.apache.org> Open Source Java <consulting, training and solutions> Sociedad Andaluza para el Desarrollo de la Sociedad de la Informaci贸n, S.A.U. (SADESI)
On Tue, 2009-11-10 at 08:40 +0100, Chapuis Bertil wrote:
> I had the same problem and solved it in my handler's implementation by only
> using local variables and limited concurrent accesses. As I understand the
> issue the same limitation could occurs with a custom CrawlingDroid
> implementation since all workers are using the same Droid. A nice fix could
> be to make the Droid and the GenericFactory abstractions clonable and
> invoking the clone method in the Worker's constructor.
Could you provide a patch? When I talked with Javier we discussed this
option as well and IMO it is an elegant solution.
Thanks for your feedback Bertil.
salu2
>
> Best regards,
>
> Bertil Chapuis
>
>
> On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler <
> thorsten.scherler.ext@juntadeandalucia.es> wrote:
>
> > On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> > > Hi, I'm working with Droids and made some URL crawlers to save a lot of
> > web
> > > pages in disk. In JUnit test, I run a little http server and crawl 20
> > pages,
> > > the most times everything works ok but in rare cases I get an error. I
> > found
> > > the problem in the HandlerFactory implementation, in the example the call
> > to
> > > handlers is like this:
> > >
> > > protected void handle(ContentEntity entity, Link link)
> > > throws DroidsException, IOException
> > > {
> > > droid.getHandlerFactory().handle(link.getURI(), entity);
> > > }
> > >
> > >
> > > If two or more workers is trying to handle at same time, the
> > HandlerFactory
> > > will handle the all with the same instance of the handler. The solution
> > > could be saving memory or improving performance.
> > >
> > > The first solution could be implemented adding a "synchronized" to
> > > HandlerFactory.handle like this.
> > >
> > > public synchronized boolean handle(URI uri, ContentEntity entity)
> > > throws DroidsException, IOException {
> > > for (Handler handler : getMap().values()) {
> > > handler.handle(uri, entity);
> > > }
> > > return true;
> > > }
> > >
> > > Only one handler to share with all workers but this solution is a
> > > performance killer. The other approx should be the opposite, each worker
> > had
> > > his own instance of
> > > the handlerfactory or handler.
> > >
> > > Solution that you think might be more appropriate?
> >
> > It depends on the usecase I guess. However I think the second option is
> > the more common solution.
> >
> > salu2
> >
> > >
> > > Salu2.
> > --
> > Thorsten Scherler <thorsten.at.apache.org>
> > Open Source Java <consulting, training and solutions>
> >
> > Sociedad Andaluza para el Desarrollo de la Sociedad
> > de la Informaci贸n, S.A.U. (SADESI)
> >
> >
> >
> >
> >
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>
Sociedad Andaluza para el Desarrollo de la Sociedad
de la Informaci贸n, S.A.U. (SADESI)
I had the same problem and solved it in my handler's implementation by only
using local variables and limited concurrent accesses. As I understand the
issue the same limitation could occurs with a custom CrawlingDroid
implementation since all workers are using the same Droid. A nice fix could
be to make the Droid and the GenericFactory abstractions clonable and
invoking the clone method in the Worker's constructor.
Best regards,
Bertil Chapuis
On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler <
thorsten.scherler.ext@juntadeandalucia.es> wrote:
> On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> > Hi, I'm working with Droids and made some URL crawlers to save a lot of
> web
> > pages in disk. In JUnit test, I run a little http server and crawl 20
> pages,
> > the most times everything works ok but in rare cases I get an error. I
> found
> > the problem in the HandlerFactory implementation, in the example the call
> to
> > handlers is like this:
> >
> > protected void handle(ContentEntity entity, Link link)
> > throws DroidsException, IOException
> > {
> > droid.getHandlerFactory().handle(link.getURI(), entity);
> > }
> >
> >
> > If two or more workers is trying to handle at same time, the
> HandlerFactory
> > will handle the all with the same instance of the handler. The solution
> > could be saving memory or improving performance.
> >
> > The first solution could be implemented adding a "synchronized" to
> > HandlerFactory.handle like this.
> >
> > public synchronized boolean handle(URI uri, ContentEntity entity)
> > throws DroidsException, IOException {
> > for (Handler handler : getMap().values()) {
> > handler.handle(uri, entity);
> > }
> > return true;
> > }
> >
> > Only one handler to share with all workers but this solution is a
> > performance killer. The other approx should be the opposite, each worker
> had
> > his own instance of
> > the handlerfactory or handler.
> >
> > Solution that you think might be more appropriate?
>
> It depends on the usecase I guess. However I think the second option is
> the more common solution.
>
> salu2
>
> >
> > Salu2.
> --
> Thorsten Scherler <thorsten.at.apache.org>
> Open Source Java <consulting, training and solutions>
>
> Sociedad Andaluza para el Desarrollo de la Sociedad
> de la Informaci髇, S.A.U. (SADESI)
>
>
>
>
>
On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> Hi, I'm working with Droids and made some URL crawlers to save a lot of web
> pages in disk. In JUnit test, I run a little http server and crawl 20 pages,
> the most times everything works ok but in rare cases I get an error. I found
> the problem in the HandlerFactory implementation, in the example the call to
> handlers is like this:
>
> protected void handle(ContentEntity entity, Link link)
> throws DroidsException, IOException
> {
> droid.getHandlerFactory().handle(link.getURI(), entity);
> }
>
>
> If two or more workers is trying to handle at same time, the HandlerFactory
> will handle the all with the same instance of the handler. The solution
> could be saving memory or improving performance.
>
> The first solution could be implemented adding a "synchronized" to
> HandlerFactory.handle like this.
>
> public synchronized boolean handle(URI uri, ContentEntity entity)
> throws DroidsException, IOException {
> for (Handler handler : getMap().values()) {
> handler.handle(uri, entity);
> }
> return true;
> }
>
> Only one handler to share with all workers but this solution is a
> performance killer. The other approx should be the opposite, each worker had
> his own instance of
> the handlerfactory or handler.
>
> Solution that you think might be more appropriate?
It depends on the usecase I guess. However I think the second option is
the more common solution.
salu2
>
> Salu2.
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>
Sociedad Andaluza para el Desarrollo de la Sociedad
de la Informaci贸n, S.A.U. (SADESI)
Dear Droids Developers,
This email was sent by an automated system on behalf of the Apache Incubator PMC.
It is an initial reminder to give you plenty of time to prepare your quarterly
board report.
The board meeting is scheduled for Wed, 18 November 2009, 2 pm Pacific. The report
for your podling will form a part of the Incubator PMC report. The Incubator PMC
requires your report to be submitted one week before the board meeting, to allow
sufficient time for review.
Please submit your report with sufficient time to allow the incubator PMC, and
subsequently board members to review and digest. Again, the very latest you
should submit your report is one week prior to the board meeting.
Thanks,
The Apache Incubator PMC
Submitting your Report
----------------------
Your report should contain the following:
* Your project name
* A brief description of your project, which assumes no knowledge of the project
or necessarily of its field
* A list of the three most important issues to address in the move towards
graduation.
* Any issues that the Incubator PMC or ASF Board might wish/need to be aware of
* How has the community developed since the last report
* How has the project developed since the last report.
This should be appended to the Incubator Wiki page at:
http://wiki.apache.org/incubator/November2009
Note: This manually populated. You may need to wait a little before this page is
created from a template.
Mentors
-------
Mentors should review reports for their project(s) and sign them off on the
Incubator wiki page. Signing off reports shows that you are following the
project - projects that are not signed may raise alarms for the Incubator PMC.
Incubator PMC
Every month, a third of incubator podlings must submit reports to the incubator PMC. I have written and am about to test a script that will run at the beginning of each month to send out reminders to those podlings that are due to report. As your report is due this month, if everything goes to plan you'll see a reminder mail soon after this one. When you do, please let me know if you see any errors. Please note, I'm likely not subscribed to this list. Upayavira
Hi, I'm working with Droids and made some URL crawlers to save a lot of web
pages in disk. In JUnit test, I run a little http server and crawl 20 pages,
the most times everything works ok but in rare cases I get an error. I found
the problem in the HandlerFactory implementation, in the example the call to
handlers is like this:
protected void handle(ContentEntity entity, Link link)
throws DroidsException, IOException
{
droid.getHandlerFactory().handle(link.getURI(), entity);
}
If two or more workers is trying to handle at same time, the HandlerFactory
will handle the all with the same instance of the handler. The solution
could be saving memory or improving performance.
The first solution could be implemented adding a "synchronized" to
HandlerFactory.handle like this.
public synchronized boolean handle(URI uri, ContentEntity entity)
throws DroidsException, IOException {
for (Handler handler : getMap().values()) {
handler.handle(uri, entity);
}
return true;
}
Only one handler to share with all workers but this solution is a
performance killer. The other approx should be the opposite, each worker had
his own instance of
the handlerfactory or handler.
Solution that you think might be more appropriate?
Salu2.
Team,
For those Lucene fanatics not in Oakland this week for ApacheCon US,
don't miss the FREE live video streaming, starting today:
http://streaming.linux-magazin.de/en/program-apachecon-us-2009.htm
Note that there are many talks available, covering Apache Hadoop,
Apache HTTPD, Lucene, as well as the Apache Pioneer's Panel and
keynote presentations.
Lucene's track is this Friday (NOTE these times are UTC -- use
http://www.timeanddate.com to map to your time zone):
17:00 Implementing an Information Retrieval Framework for an
Organizational Repository, Sithu D Sudarsan
18:00 Apache Mahout - Going from raw data to information
Isabel Drost
19:15 MIME Magic with Apache Tika
Jukka Zitting
20:15 Keynote: How Open Source Developers Can (Still!) Save The World
Brian Behlendorf
22:00 Building Intelligent Search Applications with the Lucene
Ecosystem, Ted Dunning
23:00 Realtime Search
Jason Rutherglen
Happy viewing,
Mike
On Nov 3, 2009, at 3:34 AM, Thorsten Scherler wrote: > The droids pmc proposes Javier Purto to become a new committer. > +1
On Tue, 2009-11-03 at 09:34 +0100, Thorsten Scherler wrote: > The droids pmc proposes Javier Purto to become a new committer. Upss sorry Javier! It is Puerto. Javier Puerto! sorry! salu2 > > He has offered great contributions since the beginning of the project > (even in labs) follows them through with the feeback given from the > developers. > > Please cast your votes. The voting period will end a week from today. > > http://www.timeanddate.com/counters/customcounter.html?year=2009&month=11&day=10 > > salu2 -- Thorsten Scherler <thorsten.at.apache.org> Open Source Java <consulting, training and solutions> Sociedad Andaluza para el Desarrollo de la Sociedad de la Informaci贸n, S.A.U. (SADESI)
On Tue, 2009-11-03 at 09:34 +0100, Thorsten Scherler wrote: > The droids pmc proposes Javier Purto to become a new committer. +1 salu2 -- Thorsten Scherler <thorsten.at.apache.org> Open Source Java <consulting, training and solutions> Sociedad Andaluza para el Desarrollo de la Sociedad de la Informaci贸n, S.A.U. (SADESI)
The droids pmc proposes Javier Purto to become a new committer. He has offered great contributions since the beginning of the project (even in labs) follows them through with the feeback given from the developers. Please cast your votes. The voting period will end a week from today. http://www.timeanddate.com/counters/customcounter.html?year=2009&month=11&day=10 salu2
I did some research to find the ASF projects that manage their websites with Apache Forrest, and am sending similar email to each project's dev mail list. The purposes of this email are to remind people about some of the useful facilities of Forrest, and also alert them to discussion about the status and future directions of Forrest, and to appeal for people to assist Forrest. --- oOo --- These are useful facilities to assist with developing and managing a Forrest solution for your project's website. "How to deploy documentation with the Forrestbot svn workstage" This explains how the Forrest project manages our own documentation. http://forrest.apache.org/howto-forrestbot-svn.html "Generate an ASF mirrors page using interactive web form" http://forrest.apache.org/docs/dev/howto/howto-asf-mirror.html "ForrestBar - Firefox toolbar to ease navigation and search of Forrest resources" http://forrest.apache.org/tools/forrestbar.html "How to do development with Apache Forrest" http://forrest.apache.org/howto-dev.html "Frequently Asked Questions" http://forrest.apache.org/faq.html "The Anakia output plugin" This was developed to assist the old Incubator website to stop using Forrest and export all content to an Anakia "xdoc" format. From there it could used by an Anakia-based build system, or be further transformed. http://forrest.apache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.output.Anakia/ As usual, if you need further assistance with anything then please ask on the Forrest mail lists. --- oOo --- There is discussion currently underway on the Forrest dev mail list about the current status and future direction of Forrest. http://thread.gmane.org/gmane.text.xml.forrest.devel/27325 If anyone can assist Forrest, in any capacity, then please do. -David
[ https://issues.apache.org/jira/browse/DROIDS-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorsten Scherler closed DROIDS-67.
-----------------------------------
Resolution: Fixed
thanks javier
> Better control of the finished task
> -----------------------------------
>
> Key: DROIDS-67
> URL: https://issues.apache.org/jira/browse/DROIDS-67
> Project: Droids
> Issue Type: Improvement
> Components: core
> Reporter: Javier Puerto
> Attachments: awaitTermination.patch
>
>
> The interface TaskMaster defines the method:
> void awaitTermination(long timeout, TimeUnit unit) throws InterruptedException;
> But the implementation of the MultiThreadedTaskMaster is based on ExecutorService and
this have the same method returning a boolean value.
> This kind of behaviour is better because in the case of you want to made two MultiThread
Droid and one is the consumer from the other, you can't control if the first finished the
work (producer) to run the second (consumer).
> Attached is a patch that solves the problem, is implemented also on SequentialTaskMaster.
> Salu2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/DROIDS-67?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769991#action_12769991
]
Thorsten Scherler commented on DROIDS-67:
-----------------------------------------
Committed revision 829755.
> Better control of the finished task
> -----------------------------------
>
> Key: DROIDS-67
> URL: https://issues.apache.org/jira/browse/DROIDS-67
> Project: Droids
> Issue Type: Improvement
> Components: core
> Reporter: Javier Puerto
> Attachments: awaitTermination.patch
>
>
> The interface TaskMaster defines the method:
> void awaitTermination(long timeout, TimeUnit unit) throws InterruptedException;
> But the implementation of the MultiThreadedTaskMaster is based on ExecutorService and
this have the same method returning a boolean value.
> This kind of behaviour is better because in the case of you want to made two MultiThread
Droid and one is the consumer from the other, you can't control if the first finished the
work (producer) to run the second (consumer).
> Attached is a patch that solves the problem, is implemented also on SequentialTaskMaster.
> Salu2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/DROIDS-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorsten Scherler reopened DROIDS-65:
-------------------------------------
sonatype does not response to the above search, but if you browse the rep in the tree view
you will find a couple of artifacts. However not all artifacts are there. This issue needs
a little more investigation.
> Add Droids to ASF Maven repository
> ----------------------------------
>
> Key: DROIDS-65
> URL: https://issues.apache.org/jira/browse/DROIDS-65
> Project: Droids
> Issue Type: Improvement
> Components: infrastructure
> Reporter: Otis Gospodnetic
> Priority: Minor
>
> Droids should be added to Apache's Maven repository: https://repository.apache.org/index.html#nexus-search;droids
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/DROIDS-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorsten Scherler closed DROIDS-65.
-----------------------------------
Resolution: Fixed
When all the mirrors are updated droids will be listed in the snapsshots rep.
> Add Droids to ASF Maven repository
> ----------------------------------
>
> Key: DROIDS-65
> URL: https://issues.apache.org/jira/browse/DROIDS-65
> Project: Droids
> Issue Type: Improvement
> Components: infrastructure
> Reporter: Otis Gospodnetic
> Priority: Minor
>
> Droids should be added to Apache's Maven repository: https://repository.apache.org/index.html#nexus-search;droids
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/DROIDS-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767954#action_12767954
]
Thorsten Scherler commented on DROIDS-65:
-----------------------------------------
Committed revision 827812.
You have to add the following to your settings.xml to be able to deploy the snapshot:
<settings>
<servers>
<server>
<id>org.apache.people</id>
<directoryPermissions>775</directoryPermissions>
<filePermissions>644</filePermissions>
<privateKey>/Users/YOU/.ssh/id_rsa</privateKey>
<passphrase>yourPhrase</passphrase>
</server>
</servers>
</settings>
You need to be committer to be able to make the deploy.
> Add Droids to ASF Maven repository
> ----------------------------------
>
> Key: DROIDS-65
> URL: https://issues.apache.org/jira/browse/DROIDS-65
> Project: Droids
> Issue Type: Improvement
> Components: infrastructure
> Reporter: Otis Gospodnetic
> Priority: Minor
>
> Droids should be added to Apache's Maven repository: https://repository.apache.org/index.html#nexus-search;droids
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[ https://issues.apache.org/jira/browse/DROIDS-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Javier Puerto updated DROIDS-67:
--------------------------------
Attachment: awaitTermination.patch
Changes the interface TaskMaster and the classes that implement it to support the boolean
return type on the awaitTermination method.
> Better control of the finished task
> -----------------------------------
>
> Key: DROIDS-67
> URL: https://issues.apache.org/jira/browse/DROIDS-67
> Project: Droids
> Issue Type: Improvement
> Components: core
> Reporter: Javier Puerto
> Attachments: awaitTermination.patch
>
>
> The interface TaskMaster defines the method:
> void awaitTermination(long timeout, TimeUnit unit) throws InterruptedException;
> But the implementation of the MultiThreadedTaskMaster is based on ExecutorService and
this have the same method returning a boolean value.
> This kind of behaviour is better because in the case of you want to made two MultiThread
Droid and one is the consumer from the other, you can't control if the first finished the
work (producer) to run the second (consumer).
> Attached is a patch that solves the problem, is implemented also on SequentialTaskMaster.
> Salu2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Better control of the finished task
-----------------------------------
Key: DROIDS-67
URL: https://issues.apache.org/jira/browse/DROIDS-67
Project: Droids
Issue Type: Improvement
Components: core
Reporter: Javier Puerto
The interface TaskMaster defines the method:
void awaitTermination(long timeout, TimeUnit unit) throws InterruptedException;
But the implementation of the MultiThreadedTaskMaster is based on ExecutorService and this
have the same method returning a boolean value.
This kind of behaviour is better because in the case of you want to made two MultiThread Droid
and one is the consumer from the other, you can't control if the first finished the work (producer)
to run the second (consumer).
Attached is a patch that solves the problem, is implemented also on SequentialTaskMaster.
Salu2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Luk谩拧 Vl膷ek wrote: > Hi, > > I just noticed that Niocchi has been released recently. > http://www.niocchi.com/ > > Niocchi is a java asynchronous crawl library implemented with NIO. It is > designed to crawl several thousands of hosts in parallel on a single low > end server.It is currently being used in production by Enormo > <http://www.enormo.com/> to crawl thousands of websites daily, and > by Vitalprix <http://www.vitalprix.com/>. Well, of course we should optimize our use of resources, and we could check what this library can offer - but I doubt that optimizations on this level would bring significant benefits in terms of increased speed of crawling - low-level IO handling is rarely the bottleneck. Most of the time the politeness limits (max rate of requests per host) are the bottleneck. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
Hi, I just noticed that Niocchi has been released recently. http://www.niocchi.com/ Niocchi is a java asynchronous crawl library implemented with NIO. It is designed to crawl several thousands of hosts in parallel on a single low end server.It is currently being used in production by Enormo<http://www.enormo.com/> to crawl thousands of websites daily, and by Vitalprix<http://www.vitalprix.com/> . Regards, Lukas
Hy Droids, I found the OMAS (Open Multi-Agent System) platform that have some cool feature (but works only on windows/mac). Description : "OMAS is a multi-agent platform which allows prototyping systems of cognitive agents" I recommend to read this : Main features : http://www.utc.fr/%7Ebarthes/OMAS/N244L-OMAS-7-Features.pdf For further inspection : official web-site : http://www.utc.fr/~barthes/OMAS/ I don't well know if multi-agent (+ cognitive?) is on the target of droids, but maybe it gave some idea to someone... ++
Create DOAP for Droids
----------------------
Key: DROIDS-66
URL: https://issues.apache.org/jira/browse/DROIDS-66
Project: Droids
Issue Type: Improvement
Components: infrastructure
Reporter: Otis Gospodnetic
Priority: Minor
DOAP for Droids should be created and added to http://projects.apache.org/indexes/alpha.html#D
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Add Droids to ASF Maven repository
----------------------------------
Key: DROIDS-65
URL: https://issues.apache.org/jira/browse/DROIDS-65
Project: Droids
Issue Type: Improvement
Components: infrastructure
Reporter: Otis Gospodnetic
Priority: Minor
Droids should be added to Apache's Maven repository: https://repository.apache.org/index.html#nexus-search;droids
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Just a friendly reminder to all about Lucene ecosystem events at ApacheCon US this year. We have two days of talks on pretty much every project under Lucene (see http://lucene.apache.org/#14+August+2009+-+Lucene+at+US+ApacheCon ) plus a meetup and a two day training on Lucene and a 1 day training on Solr. The Lucene training will cover Lucene 2.9 and I'm sure Erik's Solr one will cover Solr 1.4. I also know there will be quite a few Lucene, et. al. committers at ApacheCon this year, so it should be a good year to interact and discuss your favorite projects. ApacheCon US is in Oakland (near San Francisco) the week of November 2nd. The trainings are on the 2nd and 3rd, and the main conference starts on the 4th. You can register at http://www.us.apachecon.com/c/acus2009/ Hope to see you there, Grant