<= meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8">

Ok thank you Andi.

I=E2=80=99ll use the sidepath with the bytes for the moment.

Hope it will= get solved soon though.

&n= bsp;

Met vriendelijke groeten,
Marc Jeuris= sen

Bibliotheek UAntwerpen
Stadscampus =E2=80=93 Ve35.303Venusstraat 35 =E2=80=93 2000 Antwerpen
marc.jeurissen@uantwerpen.be
T +32 3 265 49 71<= /o:p>

From: Andi Vajda
Sent: woensdag 9 o= ktober 2019 23:33
To: Andi Vaj= da
Cc: pyluc= ene-dev@lucene.apache.org
Subject: Re: Field.setStringValue

On Wed, 9 Oct 2019, Andi Vajda wro= te:

&g= t;

> On Wed, 9 Oct 2019, Marc J= eurissen wrote:

>> Good day to you,

>> I have the following issue when setting the= value of a field, value

>> containing a ch= aracter > 160 (Pylucene 8.1.1, Python 3.7.2)

&g= t;>

>> ...

>= ;> (Pdb) field

>> <Field: stored,index= ed,tokenized,omitNorms

>> indexOptions=3DDO= CS_AND_FREQS_AND_POSITIONS_AND_OFFSETS<text:>>

>> (Pdb) value =3D '=C2=ABVolgende facturen werden verstuurd aan= de financi=C3=ABle

>> dienst.=C2=BB'

>> (Pdb) type(value)

&g= t;> <class 'str'>

>> (Pdb) field.se= tStringValue(value)

>> (Pdb) field

>> <Field:

>> sto= red,indexed,tokenized,omitNorms,indexOptions=3DDOCS_AND_FREQS_AND_POSITIONS_= AND_OFFSETS<text:=C2=ABVolgende

>> factu= ren werden verstuurd aan de financi=C3=ABle dienst>>

>> The field value has lo= st 2 characters.

>> But when I encode value:

>> (Pdb) value =3D value.encode('utf-8')

>> (Pdb) value

>> b= '\xc2\xabVolgende facturen werden verstuurd aan de financi\xc3\xable

>> dienst.\xc2\xbb'

>&= gt;

>> (Pdb) field.setStringValue(value)

>> (Pdb) field

>&= gt; <Field:

>> stored,indexed,tokenized,= omitNorms,indexOptions=3DDOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS<text:=C2= =ABVolgende

>> facturen werden verstuurd aa= n de financi=C3=ABle dienst.=C2=BB>>

>>= ;

>> The field value is correct.

>> So what does fi= eld.setStringValue expect: a string (as says the Lucene

>> documentation) or a byte sequence?

> Indeed, there is a pro= blem. I was able to reproduce it with just

> S= tringBuffer, no lucene involved at all:

>&= nbsp;

>>>> from lucene import in= itVM

>>>> initVM()

>>>> b=3Db'\xc2\xabVolgende facturen werden verstuurd aan= de

>>>> financi\xc3\xabledienst.\xc2= \xbb'

>>>> a=3Db.decode('utf-8')

>>>> from java.lang import StringBuffer

>>>> StringBuffer(b)

> <StringBuffer: =C2=ABVolgende facturen werden verstuurd aan de= financi=C3=ABledienst.=C2=BB>

>>>>= StringBuffer(a)

> <StringBuffer: =C2=ABVolg= ende facturen werden verstuurd aan de financi=C3=ABledienst>

>>>> StringBuffer(a).length()

> 59

>>>> StringBuffer(b).lengt= h()

> 61

>>>= > type(a)

> <class 'str'>

>>>> type(b)

> <clas= s 'bytes'>

> There must be a bug in the Python 'str' -> Java 'String'= conversion code.

> Any Java API such as field.= setStringValue() that expects a java.lang.String()

> can be passed a 'str' or 'bytes', JCC auto-converts as needed. This i= s very

> likely where the bug is.

Digging a bit furthe= r, it doesn't seem to be a problem when using Python 2.

I'm not implying this is a python bug, strings are just very differen= t

between python 2 and 3.

Andi..

> Andi..

>> Th= ank you very much.

>> Met vriendelijke groeten,<= /p>

>> Marc Jeurissen

= >>

>> Bibliotheek UAntwerpen

>> Stadscampus ? Ve35.303

= >> Venusstraat 35 ? 2000 Antwerpen

>> <= a href=3D"mailto:marc.jeurissen@uantwerpen.be">marc.jeurissen@uantwerpen.be<= /a>

>> T +32 3 265 49 71