lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject RE: New FieldSortedHitQueue uses Java 1.4 feature
Date Mon, 08 Mar 2004 12:19:58 GMT
Option 1) below looks better to me.

I will assume your sleuthing was correct, add it to the Lucene FAQ.

Here it is: http://www.jguru.com/forums/view.jsp?EID=1152237

Otis



--- Jamie M <jamienz@yahoo.com> wrote:
> 
> I agree entirely - the problem is with ibm's jitc, not
> lucene - I'm just not sure how likely, or how quickly,
> it would be for ibm to fix this jdk bug.
> 
> I've searched for details on ibm 1.3 jit bugs, and it
> seems like there are a few, and that the 'fixes' seem
> to be to disable jit for the class/method giving
> problems (there don't seem to be many cases of 'this
> will be fixed soon').
> 
> I've spent some time debugging this and narrowed this
> problem to a jit inlining problem with:
> org.apache.lucene.store.OutputStream#writeInt(int)
> 
> this method looks like:
>   public final void writeInt(int i) throws IOException
> {
>     writeByte((byte)(i >> 24));
>     writeByte((byte)(i >> 16));
>     writeByte((byte)(i >>  8));
>     writeByte((byte) i);
>   }
> 
> And I've come up with two workarounds for this problem
> under an ibm 1.3.1 jvm:
> 
> 1) disable jit compliation for this method by setting
> this environment variable:
> JITC_COMPILEOPT=SKIP{org/apache/lucene/store/OutputStream}{writeInt}
> 
> 2) change the method to explicitly inline the method
> calls:
>   public final void writeInt(int i) throws IOException
> {
> 	
>     if (bufferPosition >= BUFFER_SIZE)
>       flush();
>     buffer[bufferPosition++] = (byte)(i >> 24);
> 
>     if (bufferPosition >= BUFFER_SIZE)
>       flush();
>     buffer[bufferPosition++] = (byte)(i >> 16);
> 
>     if (bufferPosition >= BUFFER_SIZE)
>       flush();
>     buffer[bufferPosition++] = (byte)(i >>  8);
> 
>     if (bufferPosition >= BUFFER_SIZE)
>       flush();
>     buffer[bufferPosition++] = (byte) i;
> 
>   }
> 
> I know thats ugly, and I know that this class didn't
> change when the .tis code changes were made.
> 
> I'll try and open a PMR for this.
> 
> regards,
> 
> jamie
> 
> --- Robert Engels <rengels@ix.netcom.com> wrote:
> > Would the proper course not be to get IBM to produce
> > a patch for their JDK
> > to fix the JIT bug - I have to assume there are
> > other products where the JDK
> > bug will surface...
> > 
> > -----Original Message-----
> > From: Jamie M [mailto:jamienz@yahoo.com]
> > Sent: Sunday, March 07, 2004 5:39 PM
> > To: Lucene Developers List
> > Subject: Re: New FieldSortedHitQueue uses Java 1.4
> > feature
> > 
> > 
> > Tim,
> > 
> > I think there are two issues with 1.4 which affect
> > java 1.3 compatability:
> > - this regexp stuff
> > - the recent .tis file related changes
> > (thread:
> >
>
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg04383.html,
> > bug:
> >
> http://issues.apache.org/bugzilla/show_bug.cgi?id=27408)
> > 
> > As you suggest, the regexp stuff can be worked
> > around,
> > making the code usable with a 1.3 jvm.  However, the
> > recent .tis related changes cause problems when used
> > with IBM's 1.3.1 JVM (with jit enabled, under
> > windows.. I've not tested other OSes) which means
> > you
> > can't build indexes (except tiny ones) using such a
> > jvm.
> > 
> > This doesn't appear to be Lucene's fault - it seems
> > to
> > be an IBM jdk bug, but annoyingly IBM still uses
> > 1.3.1
> > jvms in all its biggest java products (WebSphere
> > Application Developer 4 & 5, WebSphere Application
> > Server 4 & 5 (and WebSphere Portal etc which are
> > built
> > on top)) - v5.x of these products is the latest
> > version, and they have this problem with the .tis
> > related changes.  IBM has a big chunk of the java
> > market, so unfortunately IBM 1.3.1 jvms are
> > (surprisingly) widely used, certainly in larger
> > commercial environments.  And I'm not sure how
> > quickly
> > IBM would release a fix for its jvm.
> > 
> > So, if lucene 1.4 was released as 'java 1.3
> > compatible' then that will probably result in
> > ibm-product-using people reporting bugs relating to
> > building indexes. Maybe lucene 1.4 should no longer
> > officially support java 1.3.  Or, a workaround in
> > the
> > new .tis related code could be sought to preserve
> > (ibm) java 1.3 support.  Either way, I think this
> > issue may be more of a deciding factor in whether
> > lucene 1.4 should support java 1.3 or not.
> > 
> > My personal preference is that a workaround is found
> > for the ibm 1.3.1 bug, but I've not found one yet.
> > 
> > regards,
> > 
> > jamie
> > 
> > --- Tim Jones <TJones@hoovers.com> wrote:
> > > Yes, generating the exception is more expensive
> > than
> > > applying the regular
> > > expression.  However, the code isn't run that
> > often
> > > (relatively) and the
> > > results are cached, so it shouldn't make a
> > > significant difference.
> > >
> > > Just to verify the numbers, I put together a small
> > > test.  I ran 100000
> > > iterations of integer, float and string values
> > > through the two methods:
> > >
> > > +-----------------------------------+
> > > | milliseconds to execute iteration |
> > > +-----------------------------------+
> > > | value type |  regex   | exception |
> > > +------------|----------|-----------+
> > > | integer    |  0.0484  | 0.0094    |
> > > | float      |  0.1641  | 0.311     |
> > > | string     |  0.1125  | 0.4796    |
> > > +------------+----------+-----------+
> > >
> > > The timing values were calculated by taking the
> > > total time and dividing by
> > > the number of iterations.  The only case where
> > using
> > > the exception method is
> > > better is if the values are integers (in which
> > case,
> > > no exception is thrown
> > > since integer is the first test).
> > >
> > > Profiling the program confirms the results (java
> > > -Xrunhprof:cpu=samples):
> > >
> > > CPU SAMPLES BEGIN (total = 299) Fri Mar 05
> > 08:18:27
> > > 2004
> > > rank   self  accum   count trace method
> > >    1 40.80% 40.80%     122    37
> > > java.lang.Throwable.fillInStackTrace
> > >    2  8.03% 48.83%      24    50
> > > java.lang.StringBuffer.<init>
> > >    3  7.02% 55.85%      21    51
> > > java.lang.StringBuffer.expandCapacity
> > >    4  3.01% 58.86%       9    49
> > > java.lang.StringBuffer.expandCapacity
> > >    5  2.68% 61.54%       8    41
> > > java.lang.NumberFormatException.forInputString
> > >    6  2.34% 63.88%       7    31
> > > java.util.regex.Pattern.matcher
> > > ...
> > >
> > > But, if this is the only code depending on java
> > 1.4,
> > > it seems like it would
> > > be better to remove it for better version
> > > compatibility.  Perhaps what would
> > > be best would be to have the code detect which
> > > version it's running under
> > > and act appropriately.
> > >
> > > Tim
> > >
> > >
> > > > From: Mario Ivankovits [mailto:mario@ops.co.at]
> > > >
> > > > I dont know where and how often this piece of
> > code
> > > gets
> > > > called and how
> > > > often a wrong value will be passed, but to throw
> > > an exception
> > > > might be
> > > > more expensive.
> > > > Think of the stacktrace which needs to be
> > filled.
> > > >
> > > > Maybe the type of the term should then be cached
> > > (per field).
> > > >
> > > >
> > > > Mario
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > > lucene-dev-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> > > lucene-dev-help@jakarta.apache.org
> > >
> > 
> > 
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! Search - Find what youre looking for faster
> > http://search.yahoo.com
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > lucene-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > lucene-dev-help@jakarta.apache.org
> > 
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > lucene-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > lucene-dev-help@jakarta.apache.org
> > 
> 
> 
> __________________________________
> Do you Yahoo!?
> Yahoo! Search - Find what you’re looking for faster
> http://search.yahoo.com
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message