lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <>
Subject Re: svn commit: r808988 - in /lucene/solr/trunk: CHANGES.txt src/java/org/apache/solr/request/
Date Fri, 11 Sep 2009 21:06:17 GMT

: A code point (unicode character) outside of the BMP (basic
: multilingual plane, fits in 16 bits) is represented as two java chars
: - a surrogate pair.  It's a single logical character - see
: String.codePointAt().  In correct UTF-8 it should be encoded as a
: single code point... but Jetty is ignoring the fact that it's a
: surrogate pair and encoding each Java char as it's own code point...
: this is often called modified-UTF8 or CESU-8.
: So... say you have this incorrect CESU-8 that is masquerading as
: UTF-8: all is not lost.

I must be missunderstanding something still ... based on your description, 
it sounds like it shouldn't matter if the encoder knows that it's one 
logical character or not, either way it should wind up outputing the same 
number of bytes....

Except that if that were the case, why would we have had this bug in the 
first place?  clearly i'm still missunderstanding something.

: Bottom line - if we correctly encapsulate whatever the servlet
: container is writing, it's certainly possible for clients to use the
: output correctly.

I still come back to not liking that this is a hardcoded hack just for 
jetty ... it seems like it would be easy for a future version of jetty to 
change this behavior in some way that makes solr start breaking -- jetty 
could fix this bug and break solr's byte counting ... that doesn't seem 

why don't we just output the raw bytes ourselves?  the code to generate 
the byte[] was/is allready there, we're just ignoring it and only using 
the length.


View raw message