Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 18 Apr 2012 18:58:41 +0000 (UTC)
From: =?utf-8?Q?Roger_H=C3=A5kansson_=28Created=29_=28JIRA=29?=
 <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <1986596739.2076.1334775521188.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (SOLR-3375) Charset problem using HttpSolrServer
 instead of CommonsHttpSolrServer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
---------------------------------------------------------------------

                 Key: SOLR-3375
                 URL: https://issues.apache.org/jira/browse/SOLR-3375
             Project: Solr
          Issue Type: Bug
          Components: clients - java
    Affects Versions: 3.6, 4.0, 3.6.1
            Reporter: Roger H=C3=A5kansson


I've written an application which sends PDF files to Solr for indexing, but=
 I also need to index some meta-data which isn't contained inside the PDF.
I recently upgraded to 3.6.0 and when recompiling my app, I got some deprec=
ated messages which mainly was to switch from CommonsHttpSolrServer to Http=
SolrServer.

The problem I've noticed since doing this, is that all extra fields which I=
 add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn't=
 matter, anything above char 127 is sent as '?'. This was not the behaviour=
 of CommonsHttpSolrServer.


I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java which =
is:
  entity.addPart(name, new StringBody(value));

The problem is that StringBody(String text) maps to=20
  StringBody(text, "text/plain", null);
and in=20
  StringBody(String text, String mimeType, Charset charset)
we have this piece of code:
  if (charset =3D=3D null) {
     charset =3D Charset.forName("US-ASCII");
  }
  this.content =3D text.getBytes(charset.name());
  this.charset =3D charset;
So unless charset is set everything is converted to US-ASCII.


On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) there =
is this line
  parts.add(new StringPart(p, v, "UTF-8"));
which adds everything as UTF-8.


The simple solution would be to change the faulty line in HttpSolrServer.ja=
va to
  entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));

However, this doesn't work either since my tests have shown that neither Je=
tty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bit =
(8859-1 I guess).

So changing HttpSolrServer.java to
  entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")))=
;
actually gives me the same result as using CommonsHttpSolrServer.


But my investigations have shown that there is a difference in how Commons-=
HttpClient and HttpClient-4.x works.
Commons-HttpClient sends all parameters as regular POST parameters but URLE=
ncoded (/update/extract?param1=3Dvalue&param2=3Dvalue2) while
HttpClient-4.x sends them as multipart/form-data messages and I think that =
the problem is that each multipart-message should have its own charset para=
meter.

I.e HttpClient-4.x sends=20
---------------------------------------------------------------------------=
--------
--jNljZ3jE1sHG529HrzSjZWYEad-6Wu
Content-Disposition: form-data; name=3D"literal.string_txt"

=C3=83=C2=A5=C3=83=C2=A4=C3=83=C2=B6
---------------------------------------------------------------------------=
--------

But it should probably send something like this

---------------------------------------------------------------------------=
--------
--jNljZ3jE1sHG529HrzSjZWYEad-6Wu
Content-Disposition: form-data; name=3D"literal.string_txt"
Content-Type: text/plain; charset=3Dutf-8

=C3=83=C2=A5=C3=83=C2=A4=C3=83=C2=B6
---------------------------------------------------------------------------=
--------


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org