Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of yseeley@gmail.com designates
 64.233.184.200 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=NvF2TLEoYGlWeuukg8l8gOYQtOHFBRu0in3HqtqVwbi1xW5JQ44QrJJvOsmwmJyweHUd7HMiK87ASihspBakoMtrLoZsAFCJDU8LEErMW1vBUQZBGwo+gtLwRZKVpGWzvKaa5qmuJfqmxkztihtEjKIfjilTF4VW+PnngTyGVOY=
Message-ID: <c68e3917050830092760ff9829@mail.gmail.com>
Date: Tue, 30 Aug 2005 12:27:42 -0400
From: Yonik Seeley <yseeley@gmail.com>
To: java-dev@lucene.apache.org
Subject: Re: Lucene does NOT use UTF-8.
In-Reply-To: <4.2.2.20050829213725.03e6b138@mail.nacimiento.com>
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_1160_31366410.1125419262881"
References: <0A17D76B-DF8B-40B9-84F2-1A5A0C83E053@rectangular.com>
	 <8B302856-A72E-497D-9858-59C32B55B9ED@rectangular.com>
	 <p0623090abf3685b5b3f4@192.168.1.42>
	 <4F608D92-3618-48A1-849E-42D10A28BE73@rectangular.com>
	 <p06230929bf382887dcb6@192.168.1.42> <43137CE9.5060200@apache.org>
	 <4.2.2.20050829213725.03e6b138@mail.nacimiento.com>

------=_Part_1160_31366410.1125419262881
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

> How will the difference impact String memory allocations? Looking at the
> String code, I can't see where it would make an impact.=20


This is from Lucene InputStream:
public final String readString() throws IOException {
int length =3D readVInt();
if (chars =3D=3D null || length > chars.length)
chars =3D new char[length];
readChars(chars, 0, length);
return new String(chars, 0, length);
}

If you know the length in bytes, you still have to allocate that many chars=
=20
(even though the number of chars may be less than the number of bytes). Not=
=20
a big deal IMHO.

A bigger pain is on the writing side, where you can't stream things because=
=20
you don't know what the length is going to be (in either bytes *or* UTF-8=
=20
chars).

So it turns out that Java's 16 bit chars were just a waste... it's still a=
=20
multibyte format *and* it takes up more space. UTF-8 would have been nice -=
=20
no conversions necessary.


-Yonik Now hiring -- http://tinyurl.com/7m67g

------=_Part_1160_31366410.1125419262881--