incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chapuis Bertil <bchap...@agimem.com>
Subject Re: HandlerFactory fails with multithreaded implementation
Date Tue, 10 Nov 2009 07:40:36 GMT
I had the same problem and solved it in my handler's implementation by only
using local variables and limited concurrent accesses. As I understand the
issue the same limitation could occurs with a custom CrawlingDroid
implementation since all workers are using the same Droid. A nice fix could
be to make the Droid and the GenericFactory abstractions clonable and
invoking the clone method in the Worker's constructor.

Best regards,

Bertil Chapuis


On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler <
thorsten.scherler.ext@juntadeandalucia.es> wrote:

> On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> > Hi, I'm working with Droids and made some URL crawlers to save a lot of
> web
> > pages in disk. In JUnit test, I run a little http server and crawl 20
> pages,
> > the most times everything works ok but in rare cases I get an error. I
> found
> > the problem in the HandlerFactory implementation, in the example the call
> to
> > handlers is like this:
> >
> > protected void handle(ContentEntity entity, Link link)
> >     throws DroidsException, IOException
> > {
> >   droid.getHandlerFactory().handle(link.getURI(), entity);
> > }
> >
> >
> > If two or more workers is trying to handle at same time, the
> HandlerFactory
> > will handle the all with the same instance of the handler. The solution
> > could be saving memory or improving performance.
> >
> > The first solution could be implemented adding a "synchronized" to
> > HandlerFactory.handle like this.
> >
> > public synchronized boolean handle(URI uri, ContentEntity entity)
> >     throws DroidsException, IOException {
> >   for (Handler handler : getMap().values()) {
> >     handler.handle(uri, entity);
> >   }
> >   return true;
> > }
> >
> > Only one handler to share with all workers but this solution is a
> > performance killer. The other approx should be the opposite, each worker
> had
> > his own instance of
> > the handlerfactory or handler.
> >
> > Solution that you think might be more appropriate?
>
> It depends on the usecase I guess. However I think the second option is
> the more common solution.
>
> salu2
>
> >
> > Salu2.
> --
> Thorsten Scherler <thorsten.at.apache.org>
> Open Source Java <consulting, training and solutions>
>
> Sociedad Andaluza para el Desarrollo de la Sociedad
> de la InformaciĆ³n, S.A.U. (SADESI)
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message