Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Fri, 20 Apr 2012 18:00:45 +0000 (UTC)
From: "Oleg Kalnichevski (Commented) (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <127278846.10596.1334944845059.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1986596739.2076.1334775521188.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (SOLR-3375) Charset problem using HttpSolrServer
 instead of CommonsHttpSolrServer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/SOLR-3375?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13258=
424#comment-13258424 ]=20

Oleg Kalnichevski commented on SOLR-3375:
-----------------------------------------

@Roger

> But it also involves getting the guys developing HttpClient to make a cha=
nge

HttpClient supports two modes for multipart MIME messages: strict and brows=
er compatible. The code snippet you have pasted above is executed in the co=
mpatibility mode only. Common browsers include a Content-Type field in body=
 parts that represent a file upload.

Oleg
               =20
> Charset problem using HttpSolrServer instead of CommonsHttpSolrServer
> ---------------------------------------------------------------------
>
>                 Key: SOLR-3375
>                 URL: https://issues.apache.org/jira/browse/SOLR-3375
>             Project: Solr
>          Issue Type: Bug
>          Components: clients - java
>    Affects Versions: 3.6
>            Reporter: Roger H=C3=A5kansson
>            Assignee: Sami Siren
>             Fix For: 3.6.1
>
>         Attachments: SolrTest.java, commonshttpsolrserver-dump.txt, https=
olrserver-dump.txt
>
>
> I've written an application which sends PDF files to Solr for indexing, b=
ut I also need to index some meta-data which isn't contained inside the PDF=
.
> I recently upgraded to 3.6.0 and when recompiling my app, I got some depr=
ecated messages which mainly was to switch from CommonsHttpSolrServer to Ht=
tpSolrServer.
> The problem I've noticed since doing this, is that all extra fields which=
 I add is sent to the Solr server as ASCII only, i.e UTF-8/ISO-8859-1 doesn=
't matter, anything above char 127 is sent as '?'. This was not the behavio=
ur of CommonsHttpSolrServer.
> I've tracked it down to a line (271 in 3.6.0) in HttpSolrServer.java whic=
h is:
>   entity.addPart(name, new StringBody(value));
> The problem is that StringBody(String text) maps to=20
>   StringBody(text, "text/plain", null);
> and in=20
>   StringBody(String text, String mimeType, Charset charset)
> we have this piece of code:
>   if (charset =3D=3D null) {
>      charset =3D Charset.forName("US-ASCII");
>   }
>   this.content =3D text.getBytes(charset.name());
>   this.charset =3D charset;
> So unless charset is set everything is converted to US-ASCII.
> On the other hand, in CommonsHttpSolrServer.java (line 310 in 3.6.0) ther=
e is this line
>   parts.add(new StringPart(p, v, "UTF-8"));
> which adds everything as UTF-8.
> The simple solution would be to change the faulty line in HttpSolrServer.=
java to
>   entity.addPart(name, new StringBody(value,Charset.forName("UTF-8")));
> However, this doesn't work either since my tests have shown that neither =
Jetty or Tomcat recognizes the strings as UTF-8 but interprets them as 8-bi=
t (8859-1 I guess).
> So changing HttpSolrServer.java to
>   entity.addPart(name, new StringBody(value,Charset.forName("ISO-8859-1")=
));
> actually gives me the same result as using CommonsHttpSolrServer.
> But my investigations have shown that there is a difference in how Common=
s-HttpClient and HttpClient-4.x works.
> Commons-HttpClient sends all parameters as regular POST parameters but UR=
LEncoded (/update/extract?param1=3Dvalue&param2=3Dvalue2) while
> HttpClient-4.x sends them as multipart/form-data messages and I think tha=
t the problem is that each multipart-message should have its own charset pa=
rameter.
> I.e HttpClient-4.x sends=20
> -------------------------------------------------------------------------=
----------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name=3D"literal.string_txt"
> =C3=83=C2=A5=C3=83=C2=A4=C3=83=C2=B6
> -------------------------------------------------------------------------=
----------
> But it should probably send something like this
> -------------------------------------------------------------------------=
----------
> --jNljZ3jE1sHG529HrzSjZWYEad-6Wu
> Content-Disposition: form-data; name=3D"literal.string_txt"
> Content-Type: text/plain; charset=3Dutf-8
> =C3=83=C2=A5=C3=83=C2=A4=C3=83=C2=B6
> -------------------------------------------------------------------------=
----------

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org