lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: SorterTemplate.quickSort causes StackOverflowError
Date Thu, 28 Apr 2011 21:54:44 GMT
> Thanks for confirming, Javier! :)
> 
> Uwe, I assume you are referring to this line 528 in MemoryIndex?
> 
>     528     if (size > 1) ArrayUtil.quickSort(entries, termComparator);
> 
> And this funky Comparator from MemoryIndex:
> 
>     208   private static final Comparator<Object> termComparator = new
> Comparator<Object>() {
>     209     @SuppressWarnings("unchecked")
>     210     public int compare(Object o1, Object o2) {
>     211       if (o1 instanceof Map.Entry<?,?>) o1 = ((Map.Entry<?,?>)
> o1).getKey();
>     212       if (o2 instanceof Map.Entry<?,?>) o2 = ((Map.Entry<?,?>)
> o2).getKey();
>     213       if (o1 == o2) return 0;
>     214       return ((Comparable) o1).compareTo((Comparable) o2);
>     215     }
>     216   };
> 
> Will try, thanks!

Yeah, simply try with mergeSort in line 528. If that helps, this comparator
is buggy.

Uwe


> ----- Original Message ----
> > From: Uwe Schindler <uwe@thetaphi.de>
> > To: java-user@lucene.apache.org
> > Sent: Thu, April 28, 2011 5:36:13 PM
> > Subject: RE: SorterTemplate.quickSort causes StackOverflowError
> >
> > Hi Otis,
> >
> > Can you reproduce this somehow and send test code? I could look  into
> > it. I don't expect the error in the quicksort algorithm itself as this
> > one is used e.g. BytesRefHash / TermsHash, if there is a bug we would
> > have  seen it long time ago.
> >
> > I have not seen this before, but I suspect a  problem in this very
> > strange comparator in MemoryIndex (which is very broken,  if you look
> > at its code - it can compare Strings with Map.Entry and so on,
> > brrrr), maybe the comparator is not stable? In this case, quicksort
> > can  easily loop endless and stack overflow. In Lucene 3.0 this used
> > stock java  sort (which is mergesort), maybe replace the
> > ArrayUtils.quickSort my  ArrayUtils.mergeSort() and see if problem is
still
> there?
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213  Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original  Message-----
> > > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > >  Sent: Thursday, April 28, 2011 11:17 PM
> > > To: java-user@lucene.apache.org
> > >  Subject: SorterTemplate.quickSort causes StackOverflowError
> > >
> > >  Hi,
> > >
> > > I'm looking at some code that uses MemoryIndex (Lucene 3.1)  and
> > > that's exhibiting a strange behaviour - it slows down over  time.
> > > The MemoryIndex contains 1 doc, of course, and executes a set of a
> > > few thousand queries against it.  The set of queries does not
> > > change - the
> > same
> > > set of queries gets executed on all incoming  documents.
> > > This code runs very quickly..... in the beginning.   But  with time is
gets
> > > slower and slower.... and slower..... and then I get  this:
> > >
> > > 4/28/11 10:32:52 PM (S) SolrException.log :
java.lang.StackOverflowError
> > >     at
> > >
> org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:104)
> > >      at
> > >
> org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:104)
> > >      at
> > >
> > > org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:
> > > 104)
> > >
> > > I haven't profiled this code yet (remote server, firewall in
> > > between,
> > can't use
> > > YourKit...), but does the above look familiar to  anyone?
> > > I've looked at the code and obviously there is the recursive  call
> > > that's problematic here - it looks like the recursion just gets
> > > deeper and deeper
> > and
> > > "gets stuck", eventually getting too deep for  the JVM's taste.
> > >
> > > Thanks,
> > > Otis
> > > ----
> > >  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene
> > > ecosystem  search :: http://search-lucene.com/
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To  unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >  For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To  unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For  additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message