incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thorsten Scherler <thorsten.scherler....@juntadeandalucia.es>
Subject Re: HandlerFactory fails with multithreaded implementation
Date Wed, 11 Nov 2009 08:25:08 GMT
On Tue, 2009-11-10 at 08:40 +0100, Chapuis Bertil wrote:
> I had the same problem and solved it in my handler's implementation by only
> using local variables and limited concurrent accesses. As I understand the
> issue the same limitation could occurs with a custom CrawlingDroid
> implementation since all workers are using the same Droid. A nice fix could
> be to make the Droid and the GenericFactory abstractions clonable and
> invoking the clone method in the Worker's constructor.

Could you provide a patch? When I talked with Javier we discussed this
option as well and IMO it is an elegant solution.

Thanks for your feedback Bertil.

salu2

> 
> Best regards,
> 
> Bertil Chapuis
> 
> 
> On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler <
> thorsten.scherler.ext@juntadeandalucia.es> wrote:
> 
> > On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote:
> > > Hi, I'm working with Droids and made some URL crawlers to save a lot of
> > web
> > > pages in disk. In JUnit test, I run a little http server and crawl 20
> > pages,
> > > the most times everything works ok but in rare cases I get an error. I
> > found
> > > the problem in the HandlerFactory implementation, in the example the call
> > to
> > > handlers is like this:
> > >
> > > protected void handle(ContentEntity entity, Link link)
> > >     throws DroidsException, IOException
> > > {
> > >   droid.getHandlerFactory().handle(link.getURI(), entity);
> > > }
> > >
> > >
> > > If two or more workers is trying to handle at same time, the
> > HandlerFactory
> > > will handle the all with the same instance of the handler. The solution
> > > could be saving memory or improving performance.
> > >
> > > The first solution could be implemented adding a "synchronized" to
> > > HandlerFactory.handle like this.
> > >
> > > public synchronized boolean handle(URI uri, ContentEntity entity)
> > >     throws DroidsException, IOException {
> > >   for (Handler handler : getMap().values()) {
> > >     handler.handle(uri, entity);
> > >   }
> > >   return true;
> > > }
> > >
> > > Only one handler to share with all workers but this solution is a
> > > performance killer. The other approx should be the opposite, each worker
> > had
> > > his own instance of
> > > the handlerfactory or handler.
> > >
> > > Solution that you think might be more appropriate?
> >
> > It depends on the usecase I guess. However I think the second option is
> > the more common solution.
> >
> > salu2
> >
> > >
> > > Salu2.
> > --
> > Thorsten Scherler <thorsten.at.apache.org>
> > Open Source Java <consulting, training and solutions>
> >
> > Sociedad Andaluza para el Desarrollo de la Sociedad
> > de la InformaciĆ³n, S.A.U. (SADESI)
> >
> >
> >
> >
> >
-- 
Thorsten Scherler <thorsten.at.apache.org>
Open Source Java <consulting, training and solutions>

Sociedad Andaluza para el Desarrollo de la Sociedad 
de la InformaciĆ³n, S.A.U. (SADESI)





Mime
View raw message